Model Quantization 2026: Running Big AI on Small Hardware
The single biggest reason a capable open-weight model won't run on the hardware you already own is memory. A model's knowledge lives in billions of numbers — its weights — and by default each one is stored in 16 bits. Quantization is the trick of storing those same numbers in 8 bits, or 4, or fewer, cutting the memory bill by half or three-quarters with a quality cost that is often too small to notice. It is the reason a 70-billion-parameter model can run on a single consumer GPU, and why "AI on a laptop" stopped being a joke.
The problem: weights are heavy
A large language model is, at heart, a very long list of numbers. A model with 70 billion parameters stored at 16-bit precision needs about 140 GB just to hold its weights in memory — before you add the working memory for a single conversation. That does not fit on one GPU; you need several expensive cards wired together, or you rent them by the hour. Memory, not raw compute, is usually the wall you hit first. Every bit you can shave off each weight is a bit you do not have to buy, power and cool.
The trick: fewer bits per number
Quantization maps each weight from a high-precision format (16-bit floating point) onto a smaller one (an 8-bit or 4-bit integer). Instead of representing a value with thousands of possible steps, you snap it to one of 256 (INT8) or just 16 (INT4) buckets, storing a scale factor so the range is preserved. The insight that makes this work is that neural networks are remarkably tolerant of noise. The exact value of any single weight barely matters; what matters is the overall pattern across billions of them. Round them all a little, and the model still says almost exactly the same thing.
Quality: what you actually give up
Quantization is not free, but the price is smaller than intuition suggests. At 8-bit, most models are effectively indistinguishable from the original on real tasks. At 4-bit with a good method, the gap on standard benchmarks is often a point or two — invisible in most applications, occasionally noticeable on the hardest reasoning or coding problems. The loss is not uniform: it tends to show up first on long, precise chains of logic and rare knowledge, and last on everyday conversational tasks. This is why the right question is never "is quantization lossless?" but "is the loss visible on my workload?"
The formats you'll hear about
"Quantization" is a family of methods, and the names matter because they signal how the model was compressed and where it runs:
- INT8 / INT4 — the target precisions: 8-bit or 4-bit integers per weight. Most other names are recipes for getting there well.
- GPTQ & AWQ — smart post-training methods that decide how to round each weight by looking at a small calibration dataset, protecting the weights that matter most. The go-to for 4-bit GPU serving.
- GGUF — the file format behind llama.cpp and the local-AI ecosystem, offering many quantization levels (Q4, Q5, Q8…) tuned to run on CPUs, laptops and phones.
- FP8 — an 8-bit floating-point format, now hardware-accelerated on modern GPUs, increasingly used to serve frontier models cheaply at near-original quality.
- QAT vs PTQ — quantization-aware training bakes the compression in during training for the best quality at low bit-widths; post-training quantization compresses an already-trained model in minutes. PTQ is far more common because it is cheap.
You rarely choose these by hand unless you self-host. They live inside inference engines and model hubs, and they pair naturally with speculative decoding and distillation to make serving cheap. When a provider's API is startlingly inexpensive for the model size, quantization is often part of the reason.
| Precision | Memory vs 16-bit | Typical quality | Best for |
|---|---|---|---|
| 16-bit (FP16/BF16) | Baseline | Full | Training, quality-critical serving |
| 8-bit (INT8/FP8) | ~½ | Near-identical | Cheap high-quality serving |
| 4-bit (INT4) | ~¼ | Small drop | Local & single-GPU deployment |
| Below 4-bit | <¼ | Growing loss | Extreme memory limits only |
Where it helps, and where it doesn't
Quantization shines when memory is the constraint: running a bigger, smarter model on the GPU you have; fitting AI onto a laptop, phone or edge device with no cloud round-trip; or cutting the per-token serving cost of a high-traffic API. It matters less when you are already compute-bound rather than memory-bound, or when your workload is the kind of exacting reasoning where every quality point counts — there, the safer move is a higher precision or a smaller model at full precision. And a badly done 4-bit quantization can degrade a model much more than a careful one, so the method is as important as the bit-width.
Why buyers should care
Even if you never quantize a model yourself, the concept reshapes what is possible and affordable. It is what lets you run a capable model on-device for privacy or offline use, and what makes self-hosting an open-weight model for EU data residency economical rather than eye-watering. The practical advice is simple: when you evaluate an open-weight model, test the specific quantized build you intend to deploy — not the full-precision version on the leaderboard — on your own prompts. Two 4-bit builds of the same model can differ, and the only benchmark that matters is your task.
The bottom line
Quantization is the quiet compression trick that put big AI within reach of ordinary hardware. By storing each weight in fewer bits, it trades a sliver of precision for a large saving in memory, cost and energy — and modern methods have made that trade so favourable that 4-bit models are now routine. The next time you run a serious model on a single GPU or a laptop, you are almost certainly running a quantized one, and you probably can't tell.
Choosing hardware for your AI stack?
We help teams size, quantize and self-host open-weight models — balancing quality, memory and cost across Anthropic, OpenAI, Mistral and local deployments, with EU data residency handled.
Talk to an AI consultant