How Much VRAM Do You Need for Local AI Models? Tables from 7B to 70B

You need about 7 GB of VRAM to run an 8B model at 4-bit quantization (Q4) with an 8,192-token context, about 11 GB for a 14B model, about 22 GB for a 32B model and about 45 GB for a 70B model. At full 16-bit precision (FP16) the same models need roughly 18, 30, 66 and 140 GB. A longer context adds more on top.

These figures come from one formula: the model's weights, plus the KV cache that holds the conversation, plus runtime overhead. Below you will find the formula, tables for 7B to 70B models at five quantization levels and four context lengths, and notes on Apple Silicon Macs. The numbers match our LLM VRAM Calculator, so you can check any other size there.

How much VRAM each model size needs#

The table shows the total memory for typical architectures at an 8,192-token (8K) context with the default 16-bit KV cache. Q8, Q6, Q5 and Q4 refer to the common GGUF formats Q8_0, Q6_K, Q5_K_M and Q4_K_M used by llama.cpp, Ollama and LM Studio.

A model fits comfortably when the total stays under about 90% of the memory you have. Between 90% and 100% it usually loads, but there is little room left for a longer chat or for other programs that use the GPU, including your desktop.

How to calculate VRAM for any model#

A running model keeps three things in memory. You can estimate each one from numbers on the model's page or in its config.json.

weights  = parameters × bits per weight ÷ 8
KV cache = 2 × layers × KV heads × head dimension × bytes per value × context length
overhead = 1 GB + 5% of the weights
total    = weights + KV cache + overhead

Weights. Quantized formats store small scaling factors next to the weights, so they take a little more than their name suggests. Q4_K_M averages about 4.85 bits per weight, Q5_K_M 5.69, Q6_K 6.56 and Q8_0 8.5. FP16 and BF16 take 16.

KV cache. For every token in the context, each layer stores a key vector and a value vector, which is where the 2 comes from. Most current models use grouped-query attention (GQA), where several query heads share one key and value head, so the cache is several times smaller than in older models.

Overhead. The runtime needs memory for its CUDA or Metal context, compute buffers and fragmentation. One gigabyte plus 5% of the weights is a cautious allowance for llama.cpp, Ollama and LM Studio.

A worked example: Llama 3.1 8B at Q4#

Llama 3.1 8B has 32 layers, 8 KV heads and a head dimension of 128, according to its configuration file.

The weights estimate is close to the real files. The llama.cpp quantization README lists these sizes for Llama 3.1 8B:

Memory tables from 7B to 70B by context length#

Each table uses a typical architecture for its size, with a 16-bit KV cache and one conversation at a time. "Weights" is the model alone; the other columns are totals.

7B and 8B models#

Based on Llama 3.1 8B (32 layers, 8 KV heads). A 7B model such as Mistral 7B needs about 0.6 GB less at Q4 and about 1.9 GB less at FP16.

What this means: an 8 GB card runs an 8B model at Q4 with up to about 8K of context. A 12 GB card runs it at Q8, and a 16 GB card gives you room for 32K.

13B and 14B models#

Based on Qwen2.5 14B (48 layers, 8 KV heads), from its config.json.

Older models without GQA need far more memory for the same context. Llama 2 13B keeps 40 KV heads instead of 8, so its cache takes about 800 KB per token: 6.3 GB at 8K instead of 1.5 GB, for a Q4 total of about 15 GB.

32B models#

Based on Qwen2.5 32B (64 layers, 8 KV heads), from its config.json.

A 24 GB card such as an RTX 3090 or 4090 runs a 32B model at Q4 with 4K to 8K of context, with little to spare. A 32 GB card runs it at Q5 or Q6. Not every model supports a 128K context, so check the model card before you plan for it.

70B models#

Based on Llama 3.1 70B (80 layers, 8 KV heads), from its config.json.

At Q4 a 70B model needs two 24 GB cards, a 48 GB workstation card or a Mac with 64 GB or more, and it will be tight on all of them. At FP16 it needs several data-center GPUs.

Why context length changes the answer#

The weights stay the same size whatever you do with the model. The KV cache does not: it grows by the same amount for every token in the context. For Llama 3.1 70B that is 320 KB per token, so the cache takes 2.5 GB at 8K, 10 GB at 32K and 40 GB at 128K.

Runtimes reserve the cache for the full context length when they load the model, not as the chat grows. If you set a 128K context, you pay for 128K even when you type one line. Two settings help:

• Set the context you need. Ollama picks a default by VRAM as of October 2026: 4K below 24 GB, 32K between 24 and 48 GB, and 256K at 48 GB and above, according to its context length docs. You can change it with OLLAMA_CONTEXT_LENGTH or the slider in the app. In llama.cpp the flag is -c.
• Quantize the KV cache. An 8-bit cache roughly halves its size and a 4-bit cache cuts it to about a quarter, with some loss in quality. For the 70B model at 32K that means about 5.3 GB or 2.8 GB instead of 10 GB.

# Ollama: 8-bit KV cache (works only with flash attention, forced on here)
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

# llama.cpp: 8-bit keys and values, 32K context
llama-server -m model.gguf -c 32768 -ctk q8_0 -ctv q8_0

The Ollama FAQ lists f16, q8_0 and q4_0 as cache types, notes that the setting applies to all models and says Ollama turns flash attention on automatically where the hardware supports it. The llama.cpp server README documents -ctk and -ctv.

How much memory an Apple Silicon Mac can use#

A Mac with Apple Silicon has unified memory: the CPU and the GPU share one pool. That lets a Mac with 64 GB or 128 GB load models that would otherwise need several graphics cards. But macOS does not hand all of that memory to the GPU.

Apple does not publish a fixed share. The practical limit is the value Metal reports as recommendedMaxWorkingSetSize, which llama.cpp prints when it loads a model. Our calculator assumes about two thirds of memory on Macs with up to 36 GB and about three quarters on larger ones. Check the number your own Mac reports.

Your browser, editor and other apps share the same memory. Close what you do not need before you load a large model.

What to do when a model does not fit#

The calculator does not estimate speed. Speed depends mostly on memory bandwidth and on how much of the model sits on the GPU, and it varies too much between setups to predict well.

Want software to run these models? See how to run an LLM locally. These apps run the GGUF files from the tables above, and each lets you set the context length:

L Llama.cpp

Efficient inference engine for C and C++ language models.

Free

🧠 AI models

O Ollama

Run advanced language models directly on personal devices.

Free

✍️ Writing

L LMStudio

Run large language models locally for privacy and convenience.

Free

🧠 AI models

K KoboldCPP

Run advanced AI models effortlessly with a user-friendly interface.

Free + from $4.00/m

🧠 AI models

FAQ#