Evaluation · completed 2026-10-28
Hardware evaluation: small open models on one CPU server
18 open models, 444 timed requests at 1 to 5 parallel slots on a 12-vCPU server with no GPU: throughput, latency, decode speed and memory.
Problem
Clients that keep data on their own servers often have no GPU. An AI agent is only useful if it answers fast enough on the hardware they already own.
Hypothesis
A small open model (under 5B parameters) can triage an email in seconds on CPU, and parallel slots raise throughput.
Success metric
Emails per minute and time to first token at 1 to 5 parallel requests, with peak memory, on identical settings.
Result
qwen3:0.6b leads: 13.9 emails/min alone, 31.9/min at 5 in parallel. 4 models cannot serve in parallel at all.
Decision
Fine-tune the smallest fast model for the task instead of serving a larger general one; test accuracy next on the full benchmark.
Setup
One server, identical settings for every model
12 vCPU
AMD EPYC 9645
AVX-512, no GPU
31 GB
RAM
Ollama 0.40.12
llama.cpp runner, every model in Q4_K_M
444
timed requests
3 runs
Method
- For each model and N = 1 to 5, Ollama is restarted with N parallel slots and the runner must report exactly N.
- 2N copies of the same email arrive at once and N channels take the next waiting job as they finish.
- Every prompt starts with a unique tag, so no request reuses another's cached prompt.
- Temperature 0, seed 42, context 8,192, 12 threads, 180s timeout; the same JSON schema is enforced for all.
- 76 measurements per request: queue wait, time to first token, decode speed, latency, memory and CPU of the runner and of the rest of the machine.
- The test email is 425 characters (about 589 prompt tokens with the system prompt).
Single request
Emails per minute, one request at a time
The baseline a single user sees: one email in, one answer out. Hover or focus a bar for latency, time to first token and memory.
- qwen3:0.6b13.9
- OLMoE-1B-7B-0125-Instruct10.6
- qwen3.5:2b8.7
- qwen3:1.7b8.6
- granite4:tiny-h7.8
- ministral-3:3b7.5
- granite4.1:3b6.0
- SmolLM3-3B6.0
- gemma3n:e2b5.4
- gemma3:4b5.2
- qwen3:4b4.7
- granite4:micro4.3
- qwen3.5:4b4.2
- phi4-mini:3.8b4.1
- llama3.2:3b1.6
- LFM2.5-8B-A1B1.4
- gemma4:e2b1.4
- LFM2.5-2.6B1.3
Highlighted: qwen3:0.6b, the base model later fine-tuned for triage.
Parallel requests
Throughput as parallel slots increase
Emails per minute with 1 to 5 requests served at once. Darker cells are faster; a dash means the runner could not provide that many slots.
| Model | ×1 | ×2 | ×3 | ×4 | ×5 | Peak RAM at max |
|---|---|---|---|---|---|---|
| qwen3:0.6b | 13.9 | 19.0 | 23.8 | 25.9 | 31.9 | 5.5 GiB |
| OLMoE-1B-7B-0125-Instruct | 10.6 | 10.7 | 13.8 | 14.4 | 14.3 | 6.6 GiB |
| qwen3.5:2b | 8.7 | – | – | – | – | 2.3 GiB |
| qwen3:1.7b | 8.6 | 14.1 | 13.9 | 15.0 | 18.9 | 6.4 GiB |
| granite4:tiny-h | 7.8 | 9.0 | 10.1 | 10.6 | 11.3 | 5.8 GiB |
| ministral-3:3b | 7.5 | 6.6 | 10.0 | 11.5 | 11.7 | 7.6 GiB |
| granite4.1:3b | 6.0 | 7.5 | 9.3 | 10.3 | 12.4 | 5.7 GiB |
| SmolLM3-3B | 6.0 | 7.0 | 8.8 | 2.7* | 10.4 | 5.2 GiB |
| gemma3n:e2b | 5.4 | 6.6 | 8.8 | 9.2 | 9.7 | 7.7 GiB |
| gemma3:4b | 5.2 | 5.7 | 7.8 | 9.2 | 8.1 | 6.2 GiB |
| qwen3:4b | 4.7 | 7.6 | 6.6 | 9.5 | 9.8 | 9.0 GiB |
| granite4:micro | 4.3 | 6.9 | 8.5 | 11.5 | 11.3 | 5.0 GiB |
| qwen3.5:4b | 4.2 | – | – | – | – | 4.0 GiB |
| phi4-mini:3.8b | 4.1 | 9.0 | 10.1 | 11.4 | 11.4 | 7.0 GiB |
| llama3.2:3b | 1.6 | 10.8 | 11.0 | 13.2 | 13.4 | 7.0 GiB |
| LFM2.5-8B-A1B | 1.4 | – | – | – | – | 5.2 GiB |
| gemma4:e2b | 1.4 | 2.4 | 3.5 | 4.5 | 5.3 | 5.8 GiB |
| LFM2.5-2.6B | 1.3 | – | – | – | – | 1.9 GiB |
* One request in that mode hit the 180 s timeout, which holds the whole mode open and explains the drop.
Findings
What the numbers say
Small and dense wins on CPU
qwen3:0.6b answers in 4.3 s alone (first token in 0.55 s) and reaches 31.9 emails/min with 5 parallel slots in 5.5 GiB of RAM.
Parallel slots trade latency for throughput
Models that support 5 slots gain 1.4× to 8.1× throughput, while each request takes longer. Sizing is a product decision: queue length versus answer time.
Not every model can be served in parallel
qwen3.5:4b, qwen3.5:2b, LFM2.5-2.6B, LFM2.5-8B-A1B ran with a single slot only in this Ollama version, which rules them out for shared use on one server.
Parameter count is not speed
gemma4:e2b decodes at only 6.2 tokens/s, while mixture-of-experts models with about 1B active parameters decode as fast as the 0.6B dense model but need several times its memory.
Full results
Every model, single request
| Model | Family | Params (active) | Quant | Size | Latency p50 | First token | Decode | RAM | Max slots |
|---|---|---|---|---|---|---|---|---|---|
| qwen3:0.6b | Qwen3 · dense | 0.6B | Q4_K_M | 0.5 GB | 4.3 s | 0.55 s | 67.1 tok/s | 1.6 GiB | 5 |
| OLMoE-1B-7B-0125-Instruct | OLMoE · moe | 6.9B (1.3B) | Q4_K_M | 4.2 GB | 5.7 s | 1.60 s | 62.6 tok/s | 4.7 GiB | 5 |
| qwen3.5:2b | Qwen3.5 · dense | 1.9B | Q4_K_M | 1.9 GB | 6.9 s | 1.34 s | 42.5 tok/s | 2.3 GiB | 1 |
| qwen3:1.7b | Qwen3 · dense | 2.0B | Q4_K_M | 1.4 GB | 7.0 s | 1.26 s | 40.8 tok/s | 2.5 GiB | 5 |
| granite4:tiny-h | Granite 4 · moe | 6.9B (1B) | Q4_K_M | 4.2 GB | 7.7 s | 2.11 s | 44.0 tok/s | 4.5 GiB | 5 |
| ministral-3:3b | Ministral 3 · dense | 3.8B | Q4_K_M | 3.0 GB | 8.0 s | 2.21 s | 40.2 tok/s | 3.9 GiB | 5 |
| granite4.1:3b | Granite 4.1 · dense | 3.4B | Q4_K_M | 2.1 GB | 9.9 s | 2.39 s | 24.9 tok/s | 2.8 GiB | 5 |
| SmolLM3-3B | SmolLM3 · dense | 3.1B | Q4_K_M | 1.9 GB | 10.0 s | 2.07 s | 30.0 tok/s | 2.6 GiB | 5 |
| gemma3n:e2b | Gemma 3n · dense | 4.5B (2B eff) | Q4_K_M | 7.8 GB | 11.2 s | 2.13 s | 27.5 tok/s | 6.7 GiB | 5 |
| gemma3:4b | Gemma 3 · dense | 3.9B | Q4_K_M | 3.4 GB | 11.6 s | 2.45 s | 27.8 tok/s | 4.1 GiB | 5 |
| qwen3:4b | Qwen3 · dense | 4.0B | Q4_K_M | 2.5 GB | 12.7 s | 2.77 s | 22.8 tok/s | 3.8 GiB | 5 |
| granite4:micro | Granite 4 · dense | 3.4B | Q4_K_M | 2.1 GB | 13.8 s | 2.20 s | 19.1 tok/s | 2.8 GiB | 5 |
| qwen3.5:4b | Qwen3.5 · dense | 4.2B | Q4_K_M | 3.3 GB | 14.3 s | 3.19 s | 21.9 tok/s | 4.0 GiB | 1 |
| phi4-mini:3.8b | Phi-4 · dense | 3.8B | Q4_K_M | 2.5 GB | 14.8 s | 2.15 s | 15.5 tok/s | 3.7 GiB | 5 |
| llama3.2:3b | Llama 3.2 · dense | 3.2B | Q4_K_M | 2.0 GB | 36.4 s | 1.96 s | 16.1 tok/s | 3.1 GiB | 5 |
| LFM2.5-8B-A1B | LFM2.5 · moe | 8.3B (1.5B) | Q4_K_M | 5.2 GB | 43.0 s | 38.99 s | 53.9 tok/s | 5.2 GiB | 1 |
| gemma4:e2b | Gemma 4 · dense | 4.6B (2.3B eff) | Q4_K_M | 4.6 GB | 44.3 s | 1.49 s | 6.2 tok/s | 5.2 GiB | 5 |
| LFM2.5-2.6B | LFM2.5 · dense | 2.6B | Q4_K_M | 1.7 GB | 46.3 s | 39.41 s | 34.5 tok/s | 1.9 GiB | 1 |
Limits
What this evaluation does not show
- It measures speed and memory, not accuracy: every request used the same email. Accuracy is measured separately on 100 held-out emails.
- One virtual server, so neighbours on the host can add noise; CPU used by other processes was recorded for every request (median 0.7%; above 1% for 14 of 444).
- Results are for one Ollama version (0.40.12) and its default runner settings; another runtime (vLLM, llama.cpp server flags) can change the parallel numbers.