How fast is an RTX 3060 12GB with Ollama?
Fast for everything that fits in its 12 GB: from 28.5 tokens per second for a 14B model up to 77.9 for a 3B. Most people read about 5 tokens per second.
| Model | Memory | RTX 3060, tokens/s | CPU only, tokens/s |
|---|---|---|---|
| Llama 3.2 3B Q4_K_M | 3.3 GB | 77.9 | 12.8 |
| Qwen2.5 7B Instruct Q4_K_M | 5.6 GB | 49.0 | 6.4 |
| Mistral 7B Q4_K_M | 5.8 GB | 57.8 | not run |
| Llama 3 8B Instruct Q4_0 | 5.8 GB | 50.3 | not run |
| Hermes 3 8B (Llama 3.1) Q4_0 | 5.8 GB | 49.9 | 5.7 |
| Qwen2.5 Coder 14B Q4_K_M | 10.4 GB | 28.5 | 3.5 |
| gpt-oss 20B MXFP4 | 14.9 GB | 6.5 (27% CPU / 73% GPU) | 2.4 |
| DeepSeek Coder 33B Q4_0 | 21.1 GB | 2.7 (47% CPU / 53% GPU) | 1.5 |
What is the best model for an RTX 3060 12GB?
For the best answers that still run fully on the card, a 14B model. For the quickest back-and-forth, a 7B or 8B model at about 50 tokens per second.
| If you want | Run | Speed |
|---|---|---|
| The fastest replies | Llama 3.2 3B | 77.9 tokens/s |
| Everyday chat and writing | A 7B or 8B model (Qwen2.5 7B, Mistral 7B, Llama 3 8B) | 49.0 to 57.8 tokens/s |
| The best answers that still fit | A 14B model (Qwen2.5 Coder 14B for code) | 28.5 tokens/s |
| A bigger model, and you can wait | gpt-oss 20B, split with system RAM | 6.5 tokens/s |
Pick by task as well as size. Qwen2.5 Coder is tuned for code. General-purpose 14B models are about the same size, so expect similar memory use and speed, but I haven’t measured them on this card yet.
Which model should run an AI agent on an RTX 3060?
Hermes 3 8B. With Ollama 0.35.1 on this card it made 27 of 30 of an agent’s routing decisions in 0.12 seconds each and followed no instruction planted in a tool result. Gemma 4 12B got all 30, but thinking made each decision take 4.1 seconds.
Speed alone doesn’t pick an agent model: it also has to choose the right tool and ignore instructions hidden in web pages and file names. Mellum2.1 and Liquid’s d1 models, tested for a local agent compares seven models on those decisions, including decision models that answer in 42 milliseconds.
Can an RTX 3060 run a 14B model?
Yes. Qwen2.5 Coder 14B used 10.4 GB, loaded 100% on the GPU and ran at 28.5 tokens per second.
It’s close to the limit. With 1.2 GB already used by the desktop and another program, 1.8 GB was still free with the model loaded. Raise the context length or open something else that uses the GPU, and the model will start splitting.
Can an RTX 3060 run gpt-oss 20B?
It runs, but not fully on the card. It needs 14.9 GB, so Ollama put 73% on the GPU and the rest in system RAM, and it generated 6.5 tokens per second.
The first answer took 65.4 seconds, including loading it from disk. If you want gpt-oss 20B at full speed, you need a 16 GB card.
Can an RTX 3060 run a 32B model?
Only slowly. DeepSeek Coder 33B needed 21.1 GB; 53% fit on the card and it generated 2.7 tokens per second.
That’s workable for a single question you can leave running, not for a conversation. For 32B models at chat speed you need 24 GB of VRAM.
How much faster is it than a CPU?
For models that fit, the RTX 3060 was 6 to 9 times faster than this machine’s CPU. When a model has to split, the gap narrows: gpt-oss 20B was 2.7 times faster split across the card than on the CPU alone.
The CPU here is an eight-core 2012 Xeon without AVX2. Core allocation, memory bandwidth and virtualization also affect speed: our four-vCPU server with AVX2 was slower on the same 3B model, with different software and thread settings. See the server comparison and its limits.
How do I get the same numbers on my card?
Run the same script I used. It needs Python and Ollama, and prints one line per model.
$ curl -O https://raw.githubusercontent.com/Arynwood-Technology/local-ai-benchmarks/main/bench.py $ python3 bench.py llama3.2:3b qwen2.5:7b-instruct --csv my-results.csv
If your results differ, or you have a different card, send them in and they’ll be added with credit.
How I tested
Intel Xeon E5-2665 (2012, 8 cores, AVX but no AVX2), 62 GB RAM, NVIDIA GeForce RTX 3060 12 GB (driver 580.178.04), Ubuntu 24.04.5 LTS, Ollama 0.11.4 with its default 4,096-token context. Same prompt for every model, temperature 0, 200-token answers; the speed is the median of three warm runs, and CPU-only runs set num_gpu to 0. About 1.2 GB of the card’s memory was already in use by the desktop and a Stable Diffusion web UI, as on most real machines. Terminal output on this page was copied from the test machine. Full method: How we test. Raw data and the script: local-ai-benchmarks. Browse the local AI benchmark dataset on Hugging Face (CC BY 4.0).
- September 24, 2026: first published, eight models on Ollama 0.11.4.
- October 9, 2026: added which model to run for an AI agent, from the agent decision benchmark.