How we test

The test machine

The original desktop tests use one older workstation. The October 5, 2026 CPU comparison adds a virtual web server. Each dataset names its hardware class and software versions; server addresses, hostnames, accounts and provider identifiers are excluded.

How is the web server comparison tested?

Four KVM vCPUs presented as an Intel Xeon Gold 6138 at 2.00 GHz, with AVX2 and AVX-512; 7.8 GiB RAM, 4 GiB swap, no compute GPU, Ubuntu 24.04.5 LTS. Ollama 0.15.2 and a temporary 0.35.1 binary use the same cached weights, CPU-only mode, three inference threads, a 4,096-token context and one loaded model at a time.

Five US-developed 1B to 3B models completed paired speed tests. Three advertise tool support and get 13 simulated cases repeated three times, with a 512-token reply cap: choosing a tool, answering without one, using a tool result and checking for recognized deletion calls after planted instructions. The injection checker recognizes native and JSON calls; other textual attempts can pass, as the server report demonstrates. These generic cases are separate from Arynwood MCP’s live evaluations.

Early swap-growth guards interrupted runs with substantial available RAM. The final policy stops below 1.5 GiB available RAM, or above 256 MiB swap growth when available RAM is also below 2 GiB. The paired report retains completed results and lists interruptions, guard changes and omitted candidates. Swap use is machine-wide. Low priority and three inference threads do not guarantee website response time.

The versions ran sequentially on a shared virtual server. Background traffic, host contention and paging can affect results. The cold measurement is a model-unloaded request with disk caches left intact, not first-token latency. Each version has one cold request and three warm requests; no confidence intervals or website-load tests were run.

How are the agent decision tests run?

Agent tests measure the decisions a local AI agent makes around a reply, not generation speed. Chat models run tool_calling.py: 13 cases with four fake tools, 3 runs each, temperature 0, a fixed seed, an 8,192-token context and a 512-token reply cap. Two cases plant an instruction to delete a clip inside a tool result; a model fails when it calls delete_clip. They also run Arynwood MCP’s 36 live evals through its real prompts, after a short warm-up, with the eval code’s commit stated.

Decision models, which return probabilities instead of text, run decision_routing.py: the same 30 Arynwood decisions and the planted-instruction cases as typed questions, 3 runs each, with a smoke test of known answers first. Decision time is measured per request after a warm-up, and peak VRAM is the highest nvidia-smi reading minus the reading before the model loaded. d1-omni-600M’s answers, from llama.cpp support a day old, were also checked against Liquid AI’s reference code. Results: Mellum2.1 and Liquid’s d1 models, tested for a local agent.

The benchmark method

• Every model gets the same prompt: “Explain in about 150 words why someone might run a language model locally.”
• Temperature 0, a fixed seed and answers capped at 200 tokens. The seed only fixes the wording; it doesn’t change speed.
• One cold run with the model unloaded (this includes loading it from disk), then three warm runs. The published speed is the median warm generation rate.
• CPU-only runs set Ollama’s num_gpu option to 0, which puts no model layers on the GPU. The model is unloaded between CPU and GPU runs.
• Memory is what Ollama itself reports while the model is loaded. Context length, thread count and quantization are stated per dataset; longer contexts can use more memory.

The script is bench.py, one file with no dependencies beyond Python and Ollama, so anyone can repeat the test and compare.

What we publish

• The hardware class, operating system, relevant driver, model quantization, context, thread settings and Ollama version behind every number, without identifying the server
• The raw results as CSV, under CC BY 4.0, in the local-ai-benchmarks repository
• Browse the local AI benchmark dataset on Hugging Face (CC BY 4.0).
• Real terminal output, copied from the test machine, when a page shows commands
• The date each page was tested and last updated, and a list of what changed
• What we did not test, next to what we did

What we don’t do

• Present someone else’s benchmark as our own, or round a number in a product’s favor
• Accept payment for placement or for a result
• Show an illustration as if it were a screenshot. Illustrations are labelled.
• Publish AI-generated artwork as Lorelei’s own work

Disclosures

Arynwood MCP, Cutroom and Arynwood Community are Arynwood’s own products, and pages that mention them say so. Arynwood sells local AI setup services. The site footer contains one affiliate link, for VPS hosting, marked as an affiliate link.

Corrections and updates

Found a mistake, or got a very different number on similar hardware? Email lorelei@arynwood.com or submit your results. Corrections are made on the page, dated, and listed in its update notes. Pages are re-tested when a new Ollama release changes the results, and at least every quarter.