—
| Hardware | Memory | Fits? |
|---|
How the estimate works
Weights ≈ params × bits ÷ 8 (a 27B model at Q4 ≈ 13.5 GB). KV cache scales with context length (estimated per-token from model size; GQA models use less). We add ~15% runtime overhead (CUDA buffers, activations, framework). Real usage varies by architecture and runtime (llama.cpp vs vLLM) — treat results as a planning estimate, not a guarantee. Training and fine-tuning need far more.
New open models get covered every Friday on This Week in AIOps.