EuclidPRO.ai · Model training

Fine-tuning a 0.6B model for email triage, on CPU

Qwen3-0.6B trained with LoRA on 2,000 specified and validated emails in 8.6 hours on 12 CPU cores. On 100 held-out test emails, category accuracy rose from 30% to 88%.

Problem

Sales inboxes mix quotation requests, orders, invoices, complaints, spam and spoofing. Off-the-shelf small models misread them, and large models need a GPU or a cloud API the client may not allow.

Hypothesis

A tiny model trained on a precisely specified dataset can match the task on CPU, if the data teaches the hard cases explicitly.

Success metric

On a held-out test set it never saw: category accuracy, field F1, schema-valid JSON and invented values, against the same model untrained.

Result

Category accuracy 30% → 88%, field F1 62 → 88, schema-valid JSON 57% → 100%, and faster.

Decision

Carry the approach into the first AI assistant for client builds; widen the test before any client use and keep a human on the to-review queue.

Results

Fine-tuned vs base on 100 held-out emails

Same harness, system prompt, JSON schema, temperature 0 and seed for both models, served by Ollama on the same CPU server. The test emails were never shown to anyone writing training data.

88%

category accuracy

base 30%

87.8

field F1

base 61.9

100%

schema-valid JSON

base 57%

3.0 s

median time per email

base 4.0 s

Category accuracy30.0% → 88.0%
Macro F1 across 8 categories12.9% → 78.6%
Field F161.9% → 87.8%
Schema-valid JSON57.0% → 100.0%
Values invented for fields that should be empty59.5% → 5.3% · lower is better
Real values from the email in the wrong field16.7% → 7.4% · lower is better

Base Qwen3-0.6B
Fine-tuned

Fine-tuned vs base: all metrics
Model	Category	Macro F1	Field F1	Empty-field violations	Not in email	Wrong field	Valid JSON	p50
Base Qwen3-0.6B	30.0%	12.9%	61.9%	59.5%	2.5%	16.7%	57.0%	4.0 s
Fine-tuned	88.0%	78.6%	87.8%	5.3%	1.4%	7.4%	100.0%	3.0 s
Fine-tuned, no schema constraint	88.0%	83.6%	89.6%	3.0%	1.2%	6.4%	89.0%	3.0 s

Per category: recall

Recall per category, base and fine-tuned
Category	Emails	Base	Fine-tuned
Quotation request	28	71%	100%
Purchase order	22	41%	95%
Invoice	12	0%	100%
Complaint	10	0%	100%
Marketing	6	17%	17%
Spam	6	0%	33%
Other	4	0%	100%
To review	12	0%	83%

Reading the results

The base model calls 72 of 100 emails a quotation request, so its 30% mostly reflects one dominant guess, not understanding.
Of the fine-tuned model's 12 category errors, 8 sent the email to the human review queue rather than acting on it, the safe failure for an agent. Its weak spots are marketing and spam, including one spam email read as an invoice: that is the next data to add.
Without Ollama's schema constraint, category accuracy is the same and extraction slightly better (the constraint forces a key order different from training), but 11% of answers break the schema. The constraint stays on, and the next training run will use its key order.
Training-style validation emails scored higher (97% category accuracy) than the real test set, which is the expected gap between practice and exam.

Dataset design

The dataset is the product spec

The model can only learn what the data shows it, so the data was specified before a single record was written: what each email must contain, how often each hard case appears, and exactly how it must be labelled.

2,000

labelled emails

batches of 50

Exact quotas per batch

Every batch of 50 has the same make-up, so no category, difficulty or sender type can drift as batches are added.

Records per category
Quotation request	440 (11 per batch)
Purchase order	360 (9 per batch)
To review	360 (9 per batch)
Invoice	280 (7 per batch)
Complaint	200 (5 per batch)
Marketing	120 (3 per batch)
Spam	120 (3 per batch)
Other	120 (3 per batch)

Difficulty 600/800/600 (easy/medium/hard) · length 480/1040/480 (short/medium/long) · 8 sender types including spoofed clients and internal forwards

Quality gates

Evidence, verbatim. Every extracted value carries the exact text it came from, and the validator checks that it occurs in the email.
Copy, never compute. Quantities, totals and dates are copied as written; no arithmetic, so the model never learns to invent numbers.
Automated validator. Schema, quotas, length bands, every trap at least twice per batch, and duplicate bodies, subjects and document numbers across all batches.
Plan-and-review loop. Each batch was generated from a row-by-row plan and checked against it before it was accepted; judgement calls were logged and resolved one way for all batches.
No leakage. Writers worked to the spec and were never given the test set or its answers.

The 20 traps the model must learn

T01Quoted older request whose numbers must be excluded
T02Forwarded email: the contact is the original author
T03Registration, VAT and phone numbers in signatures
T04Several references; pick the main one
T05Several dates; the right one for each field
T06Key values deliberately absent
T07Sounds like an order, but is not one yet
T08Marketing written in order and invoice language
T09Spoofed sender on a lookalike domain
T10Informal writing, typos, mobile sign-offs
T11Messy item tables
T12Units that must map, never be multiplied
T13Several amounts; only the stated total
T14Prompt injection telling the AI what to output
T15Internal housekeeping email
T16Change to an existing order, same number
T17Two requests in one email, with a priority
T18Sender corrects a quantity mid-email
T19Values said to be in an attachment that is missing
T20Obsolete subject over a different request

How the records were produced. I wrote the specification, the validator and the review tooling, and ran the review. The emails themselves were written by AI models working to that spec in parallel batches, and no batch entered the dataset until it passed both checks.

Training

LoRA on CPU, built to run unattended

Configuration

Base model: Qwen3-0.6B, thinking off
Method: LoRA on every attention and MLP projection
Rank / alpha / dropout: 8 / 16 / 0.05
Trainable parameters: 5.0 M (under 1% of the model)
Learning rate: 0.0002 with 14 warm-up steps, cosine decay
Batch: 8 emails per step (gradient accumulation)
Epochs / steps: 2 / 476
Max length: 1,536 tokens; loss on the answer only
Split: 1,900 train / 100 validation, stratified
Hardware: 12 CPU cores, fp32, peak 12.52 GiB RAM
Time: 8.6 h including evaluations, 58.9 s per step
Tokens trained: 3,357,100
Stack: PyTorch 2.14.1, Transformers 5.21.0, PEFT 0.21.2

Loss over training

Training loss fell from 1.34 to 0.15; validation loss kept falling into the second epoch, so the model was still learning rather than memorising.

Training loss (20-step mean)
Validation loss

Validation points every 50 steps use 40 records; the two at the epoch ends use all 100 (0.167 → 0.154). The first 25 steps (loss above 0.6) are not shown.

Generation check on 30 validation emails at the end of each epoch
Checkpoint	Valid JSON	Category	Secondary categories	Line-item count	Fields correct
End of epoch 1 (step 238)	100%	90%	80%	83%	96%
End of epoch 2 (step 476)	100%	97%	87%	93%	98%

Delivery

From a training run to a model anyone can run

Runs unattended

A system service, not a terminal session: checkpoints every 50 steps, a clean save on stop, automatic resume after a crash, and a hard stop on a non-finite loss instead of wasted hours.

Observable

Every step logs loss, learning rate, gradient norm, tokens per second, ETA and memory; validation and sample generations are logged in full for review.

Packaged for Ollama

Merged and exported to GGUF: F16 1.1 GB, Q4_K_M 378 MB. Ollama serves it with the same chat template used in training, thinking off.

Run the full 18-model accuracy benchmark, so the fine-tuned model is ranked against every off-the-shelf option, not just its own base.
Break results down by trap type to find the hard cases that still fail, and add data for exactly those.
Train with the same key order the serving schema uses, removing the one known train/serve mismatch.
Retrain on a GPU: the same run would take minutes instead of hours, which makes data iteration cheap.
Re-test on the client's own anonymised emails before go-live, with a human approving everything in the to-review queue.