Is Ollama using your GPU?

How do I check if Ollama is using my GPU?

Load a model, then run ollama ps in another terminal. The PROCESSOR column shows where the model is running.

Send the model any prompt first, because ollama ps only lists models that are loaded right now. This is Qwen2.5 7B on the RTX 3060:

$ ollama ps
NAME                   ID              SIZE      PROCESSOR    CONTEXT    UNTIL              
qwen2.5:7b-instruct    845dbda0ea48    5.6 GB    100% GPU     4096       4 minutes from now

There are three things you can see in that column, and they mean very different speeds:

What does a split like 27%/73% CPU/GPU mean?

It means the model and its working memory didn’t fit in your free VRAM, so Ollama kept part of it in system RAM. It still answers, but the part on the CPU sets the pace.

$ ollama ps
NAME           ID              SIZE     PROCESSOR          CONTEXT    UNTIL              
gpt-oss:20b    e95023cf3b7b    14 GB    27%/73% CPU/GPU    4096       4 minutes from now

gpt-oss 20B needed 14.9 GB, more than the 11.7 GB the card had free, so 27% of it went to system RAM. It generated 6.5 tokens per second. For comparison, Qwen2.5 Coder 14B needs 10.4 GB, fits entirely and ran at 28.5. A 33B model split even further:

$ ollama ps
NAME                  ID              SIZE     PROCESSOR          CONTEXT    UNTIL              
deepseek-coder:33b    acec7c0b0fd9    21 GB    47%/53% CPU/GPU    4096       4 minutes from now

To get back to full GPU speed, pick a model that fits (see how much VRAM each size needs), close other programs that hold GPU memory, or use a shorter context.

How do I confirm it with nvidia-smi?

Run nvidia-smi while the model is loaded. If Ollama is using the card, /usr/local/bin/ollama appears in the process list holding several gigabytes.

$ nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv
$ nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
memory.used [MiB], memory.total [MiB], utilization.gpu [%]
5938 MiB, 12288 MiB, 79 %
pid, process_name, used_gpu_memory [MiB]
4505, python, 584 MiB
765682, /usr/local/bin/ollama, 4838 MiB

The two numbers won’t match exactly. For the same model, ollama ps reported 5.6 GB, which is Ollama’s own estimate including room for the context, while nvidia-smi showed what the process actually held. Run watch -n 1 nvidia-smi while an answer streams and the GPU utilization jumps.

Why is Ollama not using my GPU on Linux?

Work through these in order. The first four cover most cases.

1. The driver isn’t working. nvidia-smi has to list your card. If it prints an error, fix the NVIDIA driver first; Ollama can’t use a GPU the system can’t see.
2. The machine was suspended. After a suspend and resume, Ollama sometimes loses the GPU and falls back to the CPU. Ollama’s documentation gives this fix: sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm, then restart Ollama with sudo systemctl restart ollama.
3. Ollama started before the driver. If you installed or updated the driver after Ollama, restart the service: sudo systemctl restart ollama.
4. The model is too big. A split or 100% CPU on a large model is Ollama working as designed. Check the size against your free VRAM.
5. Something forced it to the CPU. Look for num_gpu set to 0 in your requests or app settings, and check the service environment: systemctl show ollama --property=Environment.
6. It runs in Docker. The container needs GPU access, which means the NVIDIA Container Toolkit and --gpus=all on docker run.

The server log records which GPU Ollama found when it started: journalctl -u ollama --no-pager | grep -i "inference compute" (you may need sudo). I didn’t reproduce every failure above on this machine; fixes 2 and 6 come from Ollama’s hardware documentation.

How do I force Ollama to use only the CPU?

For a single request, set the num_gpu option to 0. For the whole server, start it with CUDA_VISIBLE_DEVICES=-1.

For one request

This is how every CPU-only number on Arynwood was measured. Unload the model first so it reloads on the CPU:

$ ollama stop llama3.2:3b
$ curl -s localhost:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Why run a language model locally?",
  "stream": false,
  "options": {"num_gpu": 0}
}'

Then ollama ps confirms it:

$ ollama ps
NAME          ID              SIZE      PROCESSOR    CONTEXT    UNTIL              
hermes3:8b    4f6b83f30b62    5.0 GB    100% CPU     4096       4 minutes from now

For the whole server

Ollama’s documentation says to use an invalid GPU ID. On a systemd install, add it as an override:

$ sudo systemctl edit ollama
# add these two lines in the editor, then save:
[Service]
Environment="CUDA_VISIBLE_DEVICES=-1"
$ sudo systemctl restart ollama

How do I get it back on the GPU?

Unload the model and load it again without the CPU setting. If you set the environment variable, remove it and restart the service.

$ ollama stop llama3.2:3b
$ sudo systemctl edit ollama        # delete the CUDA_VISIBLE_DEVICES line
$ sudo systemctl restart ollama
$ ollama run llama3.2:3b "hello" && ollama ps

• September 24, 2026: first published, with output from Ollama 0.11.4.