Quantization Strategies

Why Quantize

A 7-billion-parameter model at FP16 occupies ~14 GB of memory — just the weights. A 70B model needs ~140 GB. Quantization reduces each weight from 16 bits to 4-8 bits, trading a small approximation error for a large reduction in memory and faster inference.

See notebook for an interactive walkthrough — adjust bits per weight, inject outliers, and compare RTN vs K-quant error live.

This note focuses on storing and running model weights with fewer bits, especially for inference. Distributed training can also quantize communication payloads without changing the model’s stored weights in the same way. See Quantized Communication for Distributed Training for the FP8 wire-format use case in embedding-heavy recommender training.

Two Families of Quantization

All post-training quantization methods fall into two families:

	Data-free (GGUF)	Data-dependent (GPTQ)
Approach	Quantize weights directly using block statistics	Use a calibration dataset to minimize output error
Runtime	llama.cpp / Ollama (CPU, Metal, CUDA)	ExLlama / AutoGPTQ (primarily CUDA)
Speed to quantize	Seconds	Minutes to hours
Best for	Apple Silicon, CPU inference	NVIDIA GPU inference

Within the data-free family, there are two strategies — RTN (simple) and K-Quant (smarter) — both built on the same block quantization foundation.

Data-Free Quantization (GGUF)

Block Quantization — the shared foundation

Both RTN and K-Quant divide weights into blocks (typically 32 weights) and store a shared scale factor per block. This amortizes metadata cost across the block.

Quantize: $q_{i} = round (w_{i} / s)$ . Dequantize: $\overset{w}{^}_{i} = q_{i} \times s$ .

Storage breakdown: 32 weights × 4 bits = 128 data bits + 16 bits (FP16 scale) = 144 bits total. Was 32 × 16 = 512 bits at FP16. Compression: 512 / 144 = 3.6× — not 4×, because the scale factor is the overhead cost of blocking.

Block size 32 is the empirical sweet spot: large enough to amortize the scale factor, small enough that outliers don’t distort the entire block. See the block-size explorer for an interactive tradeoff chart.

Strategy A: Round-to-Nearest (RTN)

The simplest strategy: divide by the block scale, round to the nearest integer. Used by Q4_0 and Q8_0. Fast, no calibration data needed, deterministic.

Weakness: one outlier weight dominates the block’s scale, crushing all small weights into fewer quantization levels.

With 4-bit unsigned integers, quantized values range from 0 to $2^{4} - 1 = 15$ . The scale maps the block’s maximum weight to 15:

Weights:     [0.1, 0.2, 0.15, 3.8]   (4 shown; real blocks have 32)
Scale (s):   max / 15 = 3.8 / 15 = 0.253
Quantized:   [round(0.1/0.253), round(0.2/0.253), round(0.15/0.253), round(3.8/0.253)]
           = [0, 1, 1, 15]
Dequantized: [0×0.253, 1×0.253, 1×0.253, 15×0.253]
           = [0.0, 0.253, 0.253, 3.8]
Errors:      [0.1, 0.053, 0.103, 0.0]

The outlier (3.8) is preserved perfectly, but the three small weights — which cluster between 0.1 and 0.2 — can only map to levels 0 or 1 (separated by 0.253). They all get rounded to one of two values. With 32 real weights in a block, dozens of small weights may collapse to the same quantization level.

Strategy B: K-Quant (Asymmetric Quantization)

K-Quant solves RTN’s outlier problem by adding a block minimum ( $m$ ) alongside the scale ( $s$ ), shifting quantization levels to where weights actually cluster:

$\overset{w}{^}_{i} = q_{i} \times s + m$

Consider a block where weights cluster away from zero — common in LLM layers:

Weights:     [0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2]  (8 shown)

RTN (no offset, scale = max/15 = 1.2/15 = 0.08):
  Quantized:   [6, 8, 9, 10, 11, 12, 14, 15]
  Dequantized: [0.48, 0.64, 0.72, 0.80, 0.88, 0.96, 1.12, 1.20]
  Max error:   0.04

K-Quant (offset m=0.5, scale = (1.2−0.5)/15 = 0.047):
  Quantized:   [0, 2, 4, 6, 9, 11, 13, 15]
  Dequantized: [0.50, 0.59, 0.69, 0.78, 0.92, 1.01, 1.11, 1.20]
  Max error:   0.02

Why it works: RTN wastes quantization levels 0–5 on the range [0.0, 0.4] where no weights exist — those levels are never used. K-Quant’s offset shifts level 0 to 0.5, so all 16 levels cover only the actual weight range [0.5, 1.2]. Same number of levels, half the step size, half the error.

The gain is largest when weights are not centered at zero — which is the common case in LLM attention and feedforward layers.

This is asymmetric quantization (scale + offset), not K-means clustering — the “K” in the name is historical. The cost is two parameters per block instead of one, but the error drops significantly when weights aren’t centered at zero, which is common in LLM layers.

Super-blocks: K-Quant’s metadata compression

K-Quant’s extra parameter ( $m$ ) would double the metadata overhead if stored as full FP16. The solution is a two-level hierarchy: 8 consecutive blocks (8 × 32 = 256 weights) form a super-block. Each sub-block stores a cheap 6-bit relative scale and 6-bit relative min; the super-block stores shared FP16 scale and min values, amortized over 256 weights. The 6-bit values are deltas relative to the super-block’s FP16 values, encoding only a small range rather than the full model weight range.

This hierarchy is why Q4_K_M costs 4.5 bits/weight instead of 4.0 — the extra 0.5 bits buys substantially lower error. Super-blocks exist only in K-Quant formats; RTN formats (Q4_0, Q8_0) use flat blocks with no hierarchy.

GGUF Format Table

GGUF defines a family of quantization formats spanning both strategies:

Format	Bits/wt	Strategy	Notes
Q4_0	4.0	RTN	Legacy, no offset — most lossy
Q4_1	4.5	RTN	Adds block min — better than Q4_0
Q4_K_S	4.5	K-Quant	Super-blocks, small variant
Q4_K_M	4.5	K-Quant	Recommended default
Q5_K_M	5.5	K-Quant	Good if memory allows
Q6_K	6.5	K-Quant	Near-FP16 quality
Q8_0	8.0	RTN	2× memory of Q4, negligible loss
FP16	16.0	—	No quantization

Naming convention

Q = quantized
Number = bits per weight (approximate — metadata adds fractional bits)
_0 = RTN, no offset
_1 = RTN with block minimum
_K = K-Quant (asymmetric + super-blocks)
_S / _M / _L = small / medium / large variant (more metadata = more accurate = slightly more memory)

Data-Dependent Quantization (GPTQ)

GPTQ (Generalized Post-Training Quantization) takes a fundamentally different approach from the GGUF methods above: it uses a calibration dataset to minimize quantization error layer by layer.

How GPTQ works

Feed a small dataset (128–1024 samples, typically from C4 or WikiText) through the original FP16 model.
For each layer, measure how the layer’s output changes when weights are quantized.
Use the Optimal Brain Quantizer (OBQ) algorithm: quantize weights one at a time, adjusting the remaining unquantized weights to compensate for the error introduced so far. This is based on the inverse Hessian of the layer’s loss.
The result is a set of quantized weights that produce outputs close to the original model on data similar to the calibration set.

When to choose GPTQ over GGUF

Choose GPTQ when you have an NVIDIA GPU and want the absolute best quality at 4-bit. Choose GGUF K-Quant when running on Apple Silicon or CPU — GPTQ models require a CUDA runtime and don’t run on Metal.

Quality Impact: How Much Does Quantization Hurt?

Perplexity (lower = better) on WikiText-2 for Llama 3 8B:

Format	Perplexity	Δ from FP16	Memory
FP16	6.14	—	16 GB
Q8_0	6.15	+0.01	8 GB
Q6_K	6.16	+0.02	6.5 GB
Q5_K_M	6.18	+0.04	5.5 GB
Q4_K_M	6.24	+0.10	4.5 GB
Q4_0	6.42	+0.28	4 GB
Q2_K	7.89	+1.75	2.5 GB

Perplexity — a measure of model confidence

A language model assigns probabilities to each next token. Perplexity is the exponential of the average negative log-likelihood over a test corpus. A perplexity of 6 means the model is, on average, “choosing between 6 equally likely next tokens.” Lower is better. A jump from 6.14 to 6.24 (Q4_K_M) is barely perceptible in conversation. A jump to 7.89 (Q2_K) noticeably degrades coherence.

Practical takeaway: Q4_K_M for almost everyone. Q5_K_M if you have memory headroom. Avoid Q2_K unless desperate.

Edmondo's Vault

Explorer

Quantization Strategies

Quantization Strategies

Why Quantize

Two Families of Quantization

Data-Free Quantization (GGUF)

Block Quantization — the shared foundation

Strategy A: Round-to-Nearest (RTN)

Strategy B: K-Quant (Asymmetric Quantization)

Super-blocks: K-Quant’s metadata compression

GGUF Format Table

Naming convention

Data-Dependent Quantization (GPTQ)

How GPTQ works

When to choose GPTQ over GGUF

Quality Impact: How Much Does Quantization Hurt?

See also

Graph View

Table of Contents

Backlinks