Benchmark · By Lorelei Noble · September 22, 2026

Local AI without a GPU: how fast is it, really?

“Do I need a graphics card to run AI locally?” is the first question almost everyone asks. So I measured it. I ran three common Ollama models on my workstation twice: once forced onto the CPU and once on the GPU. The CPU is an 8-core Xeon from 2012 that lacks the AVX2 instructions newer chips use to speed this up, so treat these numbers as close to a floor for a desktop-class machine.

Short answer

A 3B model is comfortable on CPU alone: 12.8 tokens per second, more than twice as fast as most people read.
A 7B model on CPU is usable but slow: 6.4 tokens per second, about reading pace. Fine for short answers, tedious for long ones.
A mid-range GPU is 6–8× faster at every size. An RTX 3060 ran the same 7B model at 49 tokens per second.

The results

Generation speed in tokens per second (higher is faster). Median of three warm runs, 200-token answers.
Model	Memory used	CPU only	RTX 3060	Speed-up
TinyLlama 1.1B (Q4_0)	0.7 GB RAM · 1.3 GB VRAM	27.1	197.4	7.3×
Llama 3.2 3B (Q4_K_M)	2.6 GB RAM · 3.3 GB VRAM	12.8	77.7	6.1×
Qwen2.5 7B Instruct (Q4_K_M)	4.8 GB RAM · 5.6 GB VRAM	6.4	48.9	7.6×

The first answer after starting a model takes longer, because the model has to load into memory. From a cold start to a finished 200-token answer:

Seconds from sending the first prompt (model not yet loaded) to a complete answer.
Model	CPU only	RTX 3060
TinyLlama 1.1B	10.2 s	2.7 s
Llama 3.2 3B	18.5 s	5.6 s
Qwen2.5 7B	31.3 s	5.6 s

What these numbers feel like

A token is roughly three-quarters of a word. Most adults read somewhere around 4 words, or about 5 tokens, per second. So:

Above ~10 tokens/s, text appears faster than you can read it. Chat feels responsive. The 3B model on CPU is here.
Around 5–7 tokens/s, text arrives at reading pace. Good for a quick question; a long document summary means a minute or more of waiting. The 7B model on CPU is here.
Above ~40 tokens/s, answers feel close to instant. Every model on the GPU is here.

Speed isn’t everything. TinyLlama is fast but noticeably weaker at following instructions. For most people starting out, a 3B model is the best balance on a CPU-only machine, and a 7B model is the step up once you have a GPU with 8 GB or more of VRAM.

Which model should you start with?

Your machine	Start with	Why
8 GB RAM, no GPU	`llama3.2:3b`	Uses about 2.6 GB. Leaves room for your browser and desktop.
16 GB+ RAM, no GPU	`llama3.2:3b`, then try `qwen2.5:7b`	7B gives better answers at reading pace. Keep 3B for quick back-and-forth.
NVIDIA GPU, 8–12 GB VRAM	`qwen2.5:7b` or another 7–8B model	Fits fully in VRAM at Q4 and runs about 50 tokens per second.

Arynwood MCP’s documented baseline is 16 GB RAM, because the app, its backend and your model run side by side. On an 8 GB machine, plain Ollama chat is the better place to start.

Test your own machine in two minutes

Install Ollama, pull a model, and run it with --verbose. Ollama prints the speed after each answer; the eval rate line is the number in the tables above.

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2:3b
ollama run llama3.2:3b --verbose
>>> Explain in about 150 words why someone might run a language model locally.
...
eval rate:            12.80 tokens/s

Have a GPU but want to see the CPU-only number? Unload the model, then ask through Ollama’s API with num_gpu set to 0, which puts zero model layers on the GPU:

ollama stop llama3.2:3b
curl -s localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Explain in about 150 words why someone might run a language model locally.",
  "stream": false,
  "options": {"num_gpu": 0}
}' | python3 -c 'import json,sys; r=json.load(sys.stdin); print(round(r["eval_count"]/r["eval_duration"]*1e9, 1), "tokens/s")'

Check it worked with ollama ps: the PROCESSOR column should say 100% CPU. Run ollama stop llama3.2:3b afterwards so your next chat loads back onto the GPU.

How I tested

CPU	Intel Xeon E5-2665 (Sandy Bridge, 2012), 8 cores / 16 threads, AVX but no AVX2
Memory	62 GB system RAM
GPU	NVIDIA GeForce RTX 3060, 12 GB, driver 580.178.04
Software	Ubuntu 24.04.5 LTS, kernel 6.8, Ollama 0.11.4, default model quantizations
Method	Same prompt, temperature 0, fixed seed, answers capped at 200 tokens. One cold run, then the median of three warm runs. CPU runs forced with `num_gpu: 0`; the model was unloaded between modes.

One machine is one data point. A modern desktop CPU with AVX2 or AVX-512 and faster memory will beat these CPU numbers, often by a wide margin, so if your processor is from the last several years, expect better. Longer prompts and longer conversations also slow things down, because the model has to process everything already in the context.

Put it to work

Once a model runs, Arynwood MCP gives it a desktop workspace: multi-persona chat, a searchable knowledge base of your own documents and connections to creative tools, all against your local Ollama. You can try the interface in your browser before installing, and the Linux setup guide covers requirements and installation.

If you’d rather have it set up for you, I do paid local AI setup on Linux: one working workflow on your hardware, a walkthrough and written notes.

Try the demo Get setup help