How Much VRAM Do You Need to Run LLMs Locally?
The primary consideration when deploying a local LLM is straightforward: will the model fit within your GPU's memory? The answer hinges on model size, quantization level, and context length. This guide provides a practical framework for determining the appropriate VRAM capacity.
How quantization affects VRAM
Quantization reduces the precision used to store model weights. Lower-bit quantization shrinks the model size and reduces VRAM consumption, albeit with a slight reduction in quality.
| Quant | Bits per weight | Typical use |
|---|---|---|
| Q8_0 | 8 | Very high quality |
| Q6_K | ~6.6 | Very good quality |
| Q5_K_M | ~5.5 | Good quality and size |
| Q4_K_M | ~4.5 | Good balance of size and quality |
| Q3_K_M | ~3.5 | Lower VRAM, more quality loss |
Q4_K_M is a popular choice when VRAM is constrained. If you have additional VRAM available, Q5 or Q6 allows you to run the same model with reduced quantization pressure.
Approximate VRAM by model size
The following figures are rough estimates for model weights alone. Actual VRAM requirements will be higher due to the memory used by the runtime, KV cache, and context processing.
| Model size | Q8_0 | Q6_K | Q4_K_M | Q3_K_M |
|---|---|---|---|---|
| 4B | ~5 GB | ~4 GB | ~3 GB | ~2.5 GB |
| 8B | ~9 GB | ~7 GB | ~5.5 GB | ~4.5 GB |
| 12B | ~13 GB | ~10 GB | ~8 GB | ~6.5 GB |
| 14B | ~16 GB | ~12 GB | ~9 GB | ~7.5 GB |
| 27B | ~30 GB | ~22 GB | ~17 GB | ~13 GB |
| 32B | ~36 GB | ~27 GB | ~20 GB | ~16 GB |
| 70B | ~80 GB | ~60 GB | ~42 GB | ~34 GB |
These are estimates rather than strict limits. Variations in model architecture and quantization formats can influence the final memory footprint.
What different amounts of VRAM can run
| VRAM | Practical range | Current examples |
|---|---|---|
| 8 GB | Small models around 4B to 9B | Gemma 4 E4B, Qwen3.5 9B |
| 12 GB | Small to mid-sized models around 9B to 14B | Gemma 4 12B, Qwen3.5 9B |
| 16 GB | 12B to 27B with lower quantization | Gemma 4 26B-A4B, Qwen3.6 27B at Q4 |
| 24 GB | 27B to 35B at Q4 to Q6 | Qwen3.8 27B, Gemma 4 31B |
| 32 GB | 27B to 35B at higher quantization | Qwen3.8 27B, Gemma 4 31B |
| 48 GB | Large dense models at lower quantization | 70B-class models at Q3 to Q4 |
| 80 GB | Large dense models at higher quantization | 70B-class models at Q4 to Q6 |
These ranges apply to models where weights can reside on the GPU. Large MoE models behave differently: while only a subset of parameters is active per token, the model must still store its full set of weights. Consequently, a model with 100B or more total parameters will not fit within a VRAM budget equal to its total parameter count simply because it has fewer active parameters.
MoE models
Mixture-of-Experts (MoE) models consist of multiple parameter groups known as experts. Only specific experts are activated for each token, potentially making inference more efficient than a dense model with the same total parameter count.
However, inactive experts remain part of the model. As a result, large MoE models can demand significantly more memory than their active parameter count implies. Very large models may necessitate multiple GPUs or offloading to system RAM.
Context length also uses VRAM
Model weights represent only part of the memory requirement. The KV cache expands as context length increases, meaning running the same model at 64K context may require substantially more VRAM than at 4K.
- Longer context consumes more VRAM.
- KV-cache precision influences memory usage.
- Batch size and concurrent users also increase memory demands.
- Reserve some VRAM for the runtime rather than filling the GPU entirely with model weights.
Practical tips
- Verify the actual size of the quantized model you intend to run.
- Do not rely solely on the model file size for VRAM requirements. Allow space for the KV cache and runtime.
- If a model does not fully fit in VRAM, parts can be offloaded to system RAM, though this typically slows inference.
- For long-context or agentic workloads, budget more VRAM than is required for the model weights alone.
- Multiple GPUs can be used to distribute a model when a single GPU lacks sufficient VRAM.
Run it on DaDesktop
There is no need to purchase a GPU to run a local LLM. DaDesktop provides a cloud desktop equipped with the necessary VRAM, enabling you to run models directly without owning the hardware.
Select the VRAM tier suitable for your model, load it, and begin usage. Enjoy a setup-free experience with no hardware purchase and no driver issues. View available GPUs for current options.