Ollama Out of Memory in 2026: VRAM, RAM, Context and Quantization Fixes

If Ollama suddenly says out of memory, the first instinct is usually to look at the model's file size and compare it with the GPU's VRAM. That is useful, but it is not enough. A local LLM needs memory for model weights, the KV cache, compute/work buffers, runtime overhead, and sometimes more than one loaded model or request.

That is why a model that appears to be a “12 GB model” can still fail on a 16 GB GPU, especially after increasing context length, enabling parallel requests, or leaving another model resident. The practical fix is usually not mysterious: reduce the memory requirement, free memory, use a smaller quantization, reduce context, or allow CPU/RAM offload.

This guide gives you a repeatable way to diagnose Ollama out-of-memory errors on NVIDIA, AMD and CPU/unified-memory systems without guessing.

What an Ollama out-of-memory error actually means

Ollama has to allocate several different kinds of memory. The model download size is only one part of the equation.

Ollama's own logs can expose this distinction. In reported troubleshooting output, the runtime has shown separate values for model weights, KV requirements, graph buffers and total required memory. That is much more useful than comparing only the model's advertised download size.

The fastest diagnosis: prove whether the problem is model size, context, or something else

1. Check what Ollama currently has loaded

ollama ps

Look at the loaded model and its processor placement. If another large model is already resident, stop it before testing the failing model. For a clean test, use a short-lived keep-alive:

ollama run YOUR_MODEL
# after testing, allow it to unload or stop the Ollama process/service if necessary

On NVIDIA, also inspect the actual device:

nvidia-smi

Do not treat GPU utilization percentage as the memory measurement. A GPU can show low utilization while still holding most of its VRAM, and memory can be allocated even when compute utilization is temporarily zero.

2. Test with a deliberately small context

Context length is one of the easiest variables to change and one of the easiest to underestimate. A 32K or 128K context can require substantially more KV-cache memory than a 4K or 8K workload.

For a diagnostic test, start small rather than immediately requesting the model's maximum context:

ollama run YOUR_MODEL

Then use a modest context setting in the client or API request. For API clients, the relevant Ollama option is commonly num_ctx. For example:

{
  "model": "YOUR_MODEL",
  "prompt": "Explain this short document.",
  "options": {
    "num_ctx": 4096
  }
}

If 4K works but 32K fails, you have strong evidence that context/KV memory is the limiting factor rather than simply the model weights.

Why context length can cause an OOM even when the model fits

The KV cache stores attention information for tokens already processed. Its memory requirement grows with the number of tokens represented by the active context. The exact size depends on the model architecture, number of layers, attention dimensions and cache precision.

This is why “the model is only 10 GB and my GPU has 12 GB” is not a safe calculation. You have almost no room left for the KV cache and execution buffers.

Long-context failures have been reported in the Ollama issue tracker, including cases where a model that loaded successfully ran out of memory only after the context grew. Those reports are useful evidence of the failure mode, but they should not be treated as universal limits because hardware, model architecture, backend and Ollama version all matter.

Fix #1: Reduce context length first

If you do not actually need 32K, 64K or 128K tokens, do not reserve them. Start at 4K or 8K, verify the application, and increase gradually.

These are starting points, not guaranteed hardware limits. A smaller model may tolerate a much larger context than a larger model on the same GPU.

Fix #2: Use a smaller or more memory-efficient quantization

For GGUF-based local inference, the quantization choice directly affects weight memory. If a Q8 model does not fit, a Q6, Q5 or Q4 variant may leave enough headroom for the KV cache and runtime buffers.

Do not choose a quantization solely because its filename is smaller. The right choice is the one that fits your actual workload while preserving acceptable output quality.

For a detailed explanation of Q4_K_M, Q5_K_M, Q6_K and Q8_0, see GGUF Quantization Explained: Q4_K_M vs Q5_K_M vs Q6_K vs Q8_0.

Fix #3: Remember that the model's download size is not its runtime requirement

A model list may show a file size that looks safely below your VRAM capacity. That does not mean the complete runtime allocation will fit.

As an example, imagine a model file is roughly 11 GB and the GPU has 16 GB of VRAM. You have only about 5 GB before considering KV cache, compute buffers, other applications, driver/runtime overhead and allocator headroom. A large context can consume that remaining space quickly.

For a more systematic pre-download estimate, use the VRAM calculation guide.

Fix #4: Reduce parallelism when serving multiple requests

One model serving one sequence and one model serving several simultaneous sequences are not the same memory workload. More active sequences can require more KV-cache storage.

If you are using Ollama behind Open WebUI, an API service or another client that can issue concurrent requests, test with a single request first. If the single-request case works but concurrent requests cause OOM, reduce concurrency before changing the model.

This matters particularly on GPUs where the model already consumes most available memory. A small amount of extra KV-cache allocation per active sequence can be enough to cross the allocation limit.

Fix #5: Unload models that are being kept alive

Ollama can keep models loaded so subsequent requests start faster. That is convenient, but it also means your next model may not have the full device available.

Use:

ollama ps

to see what is resident. If your application deliberately keeps several models loaded, treat their memory as part of the total budget. A clean troubleshooting test should eliminate that variable.

Fix #6: Use CPU/RAM offload when the model is only slightly too large

CPU+GPU hybrid inference can make a model usable when it does not completely fit in VRAM. The trade-off is that moving work between system memory and GPU memory can reduce performance, particularly when the workload is heavily dependent on GPU throughput.

This is often preferable to an outright failure when your priority is “run this model locally” rather than “keep every layer on the GPU.” It is also useful on systems with substantial RAM and limited VRAM.

Do not assume that adding system RAM magically increases GPU VRAM. Offloading works only when the runtime and backend support the required placement.

Fix #7: Check the actual backend and hardware path

On NVIDIA systems, confirm the driver can see the GPU:

nvidia-smi

On AMD systems, use the appropriate ROCm/AMD monitoring tools for your platform. In Docker, also verify that the container was started with GPU access and that the required runtime/device exposure is present.

If Ollama falls back to CPU, an OOM can look very different from a VRAM-only failure. The CPU path may have a much larger RAM budget but can be dramatically slower for workloads intended to run on the GPU.

Ollama's issue tracker contains examples of both backend-specific allocation failures and cases where a GPU was detected but no layers were actually offloaded. Treat the startup log as the source of truth for what the runtime actually selected.

How to read Ollama logs without guessing

When a model fails, capture the complete startup sequence. Useful clues include:

• Which backend was selected.
• How many model layers were offloaded.
• How much memory was allocated for weights.
• How much memory was reserved for KV/context.
• Whether graph or compute buffers were allocated.
• Which GPU received the allocation on multi-GPU systems.
• The exact CUDA, ROCm or backend error.

A message such as cudaMalloc failed: out of memory tells you allocation failed, but it does not by itself tell you whether the model weights, KV cache or temporary buffers caused the failure. Earlier memory-accounting lines are therefore important.

Multi-GPU OOM: why “I have 48 GB total” can still fail

Total VRAM across multiple GPUs is not always equivalent to one large unified VRAM pool. The runtime must place tensors according to supported splitting and backend rules, and individual allocations still have device-specific constraints.

Recent Ollama GitHub reports include multi-GPU cases where a model's total memory looked sufficient but a particular GPU allocation failed. There are also reports of models being placed differently than users expected. These are version- and configuration-sensitive issues, so do not generalize one issue report into a permanent limitation.

For multi-GPU troubleshooting, first test with one GPU and a smaller context. Then add GPUs deliberately and inspect the startup logs and device memory after each change.

Ollama OOM troubleshooting checklist

1. Run ollama ps and identify already-loaded models.
2. Check actual GPU memory with the vendor's monitoring tool.
3. Retry with a small context such as 4K.
4. Run one request at a time.
5. Check the Ollama version and update if you are on an old release.
6. Read the startup log for weight, KV and buffer allocation information.
7. Try a smaller quantization if the weights leave too little headroom.
8. Try CPU/GPU hybrid placement if the model is only slightly too large.
9. For Docker, verify GPU passthrough and the correct backend libraries.
10. For multi-GPU systems, test placement one device at a time.

Common mistakes that make OOM troubleshooting harder

“The model is smaller than my VRAM, so it must fit.”

False. Runtime memory includes more than the model file.

“GPU utilization is 0%, so the GPU is unused.”

Not necessarily. Utilization is activity over a sampling interval; VRAM allocation is a different measurement.

“I increased context because the model supports it.”

A model's maximum context is not the same thing as the context your hardware can afford at a useful performance level.

“More system RAM will fix any VRAM OOM.”

Only if the runtime can offload part of the workload to RAM. It does not increase the physical VRAM on the GPU.

“The previous Ollama version worked, so the model is broken.”

Not necessarily. Runtime allocation and backend behavior can change between releases. Test the current release and consult release notes/issues before drawing conclusions.

Security matters too: do not expose an unprotected Ollama service casually

Local AI is often chosen for privacy, but a local server can still become a security problem if you expose its API carelessly. Keep Ollama bound to localhost unless remote access is genuinely required. If you expose it on a LAN or through a reverse proxy, add authentication, network controls and HTTPS where appropriate.

This is especially important because model-loading and model-management endpoints have had security issues. For example, the GitHub Advisory Database documents a high-severity GGUF loader vulnerability affecting Ollama versions before 0.17.1. Keep the runtime updated and do not assume that “localhost” means every deployment is safe.

When the right answer is to use a smaller model

Sometimes repeated tuning is the wrong solution. If your GPU has 8 GB and the workload requires a large model plus a long context, aggressively squeezing the model into memory can produce a frustrating setup with little headroom.

Choose a smaller model when:

• You need a stable application rather than an experiment.
• Your workload routinely needs long context.
• You need multiple concurrent requests.
• CPU offload makes latency unacceptable.
• You need memory for another GPU workload such as image generation.

The best local model is not the largest one that can be made to start. It is the largest model that leaves enough memory for the context, application and workload you actually use.

FAQ

Why does Ollama say out of memory when VRAM is still free?

Free VRAM reported by a monitoring tool does not guarantee that the exact allocation Ollama needs can be satisfied. The requested allocation may require a large contiguous or backend-specific buffer, and the runtime may also need memory for KV cache, graphs and other buffers.

Does lowering context really reduce VRAM usage?

Yes, in general. The active KV cache grows with context, although the exact memory relationship depends on the model architecture and cache format. Lowering context is one of the most useful first tests.

Should I use Q4 if Q8 gives an OOM?

It can be a sensible fix when weight memory is the limiting factor. But compare the available Q4/Q5/Q6 variants for the specific model and validate output quality for your task.

Can Ollama use system RAM when VRAM is full?

Depending on the backend and model, hybrid CPU/GPU execution can place part of the workload in system memory. It is a compatibility/performance trade-off, not an extension that turns system RAM into physical VRAM.

Why does a second model trigger OOM when the first one works?

Because the first model may still be resident. The second model needs its own weights and runtime memory, and concurrent loaded models can exceed the available VRAM even when each model works individually.

What should I change first: quantization or context?

If the model already loads but crashes when processing longer prompts, reduce context first. If the model cannot load at all and the weights consume nearly all available VRAM, try a smaller quantization or partial offload.

Useful GyanAangan guides

• Ollama GPU Not Being Used: Diagnose and Fix CPU Fallback
• How Much RAM and VRAM Does a Local LLM Actually Need?
• How to Calculate VRAM Usage Before Downloading a Local AI Model
• GGUF Quantization Explained: Q4_K_M vs Q5_K_M vs Q6_K vs Q8_0
• Ollama vs llama.cpp vs LM Studio vs vLLM

Official sources and further reading

• Ollama GitHub repository
• Ollama issue tracker
• llama.cpp GitHub repository
• Ollama releases
• GitHub Advisory: Ollama GGUF loader vulnerability

Bottom line

When Ollama runs out of memory, stop thinking only in terms of model-file size. Think in terms of a memory budget: weights + KV cache + compute buffers + runtime overhead + other loaded workloads.

Start with a small context and one request, verify the backend and actual device placement, inspect the startup memory accounting, and then increase context or concurrency one variable at a time. If the model is still too large, step down the quantization or use hybrid CPU/GPU inference.

That workflow turns an “Ollama out of memory” error from a guessing game into a measurable hardware and configuration problem.