Your quantization numbers were measured in the wrong place

Draft. Every number below is measured; every limit is stated. The point is not “use our tool” — it’s five specific, checkable ways a quantization measurement lies, and one discipline that catches them.

Quantization tooling asks you to trust a number. You quantize a model, run an eval, see “1.2% perplexity increase” or “98% recovery,” and ship. The problem is that the number was almost always produced somewhere other than where the model runs in production — a different quantization backend, a proxy metric, a smaller model, a single layer in isolation — and the number does not transfer.

We built a measurement engine (it started life as a numerical-parity CI gate for model deployments) and pointed it at quantization, with one rule: always measure the model you actually ship, on the metric that matters. That rule kept catching the tooling lying — and once, our own eval. Here are five cases, each one a thing worth checking in your own pipeline.

Finding 1 — the eval distribution isn’t the serving distribution

This is the one where the tool caught us. We put Gemma 4 12B — a day-one 2026 release — through the loop: quantize, export, re-evaluate the served checkpoint against a 10% perplexity bar. The gate refused our int4-GPTQ export at +581% vs fp. Scored the same way, Google’s own QAT W4A16 checkpoint for the same model read 34,000,000 perplexity — six orders of magnitude broken. Two catastrophic verdicts, both measured on the served artifact, both replicated across two serving stacks (transformers and vLLM agree). By every rule above, publishable.

Both verdicts were wrong. One probe showed it: the “broken” QAT checkpoint answers a chat-formatted prompt perfectly (“The capital of France is Paris.”) while assigning ~−27 logprobs per token to plain prose — worse than uniform over its 262k vocab. Nothing was broken. Our eval scored raw text, and an instruction-tuned model — a chat-only-QAT’d one especially — has little raw-text language modeling left to measure. We were grading a chat model on a distribution it never serves.

Re-scored on its served distribution (200 chat-formatted samples, template masked, same bar, same code):

gemma-4-12B-it, served (vLLM) chat eval (what it serves) raw-text eval
fp 25.2 3,665
int8wo (ours, 2×) +0.7% −0.6%
int4-GPTQ (ours, 4×) +3.0% — meets the bar +581%
Google QAT W4A16 (4×) −10.4% (better than fp) +936,000%

Same artifacts, same engine, same 10% bar. On the raw-text eval the gate ships 2× int8; on the served distribution it ships 4× int4 at +2.1%. The vendor checkpoint goes from “catastrophically broken” to better than the fp model (QAT included more chat training). And the two columns together are a finding in their own right: quantization damage is distribution-dependent — int8 is robust on both, int4 preserves the well-trained head of the distribution while destroying the tails, and QAT optimizes the head having abandoned the tails by design. (One family so far; treat the mechanism as a hypothesis.)

The lesson: a quality bar is only meaningful on the distribution you serve. An off-distribution eval doesn’t just add noise — it reverses verdicts in both directions: it refuses shippable models and condemns healthy ones. The fix is boring and structural: our eval sets now accept chat pairs ({"chat": [{"user": ..., "assistant": ...}]}), scored on the assistant turn with the template masked, so the whole select → export → re-eval loop gates on what the model will actually be asked to do. The full story — including the headline we nearly published — is in the Gemma 4 case study.

Finding 2 — the backend you measure in isn’t the backend you serve in

The most common quantization split in practice: you measure with one library (it has the nice per-layer hooks and filters) and deploy with another (it’s what your serving engine loads). We measure with torchao and deploy with compressed-tensors. Same scheme name, same bit-width — different implementation.

The same int4 weight-only scheme, measured in torchao vs served via compressed-tensors, across three architecture families:

family fp int4 in torchao int4 served (compressed-tensors) gap
Qwen2.5-1.5B 10.3 21.7 20.5 −6%
Llama-arch (SmolLM2-1.7B) 8.9 34.7 23.1 −33%
Gemma-2-2B 20.5 59.0 22.2 −62%

The gap is real, it grows across families, and it is directional: torchao’s int4 consistently reads worse than the served model — dramatically so on non-Qwen (Gemma int4 measures 59 in torchao but serves at 22, a 2.7× discrepancy). In other words, the cheaper-to-measure backend systematically under-sells models it wasn’t tuned on, and you’d reject an int4 deployment that would actually have been fine. (For int8 the two backends agree to ~2%; int4 is where packing and calibration differences bite.)

The lesson: a quantization number is only valid for the exact backend that produced it, and the error is worst on the architectures your measurement backend wasn’t tuned on — precisely the ones where you have the least prior intuition to sanity-check it. If you measure in framework A and serve from framework B, re-measure in B. We made this a default step — re-evaluate the served checkpoint — and it paid for itself immediately, twice more below.

Finding 3 — “98% recovery” can be a serving no-op

SmoothQuant is a well-known technique for recovering int8 activation-quantization quality. In torchao, on Qwen2.5-1.5B, it works exactly as advertised: plain w8a8 wrecks the model (18.1 perplexity vs 9.4 fp), and SmoothQuant brings it back to 9.7 — a near-full recovery.

We wired the same SmoothQuant into the served (compressed-tensors) export, expecting the same recovery. Instead, across all three families:

w8a8, served (compressed-tensors) plain + SmoothQuant torchao’s recovery for reference
Qwen2.5-1.5B 18.71 18.71 18.6 → 10.6 (−43%)
Llama-arch 52.85 52.85 52.4 → 22.4 (−57%)
Gemma-2-2B 19.80 19.80 — (w8a8 already fine)

Bit-identical, every family. SmoothQuant did nothing when served — the calibration ran, the smoothing was applied to every layer (we checked the logs), and the output was unchanged — even where torchao showed it recovering 43–57%. The reason is mechanical: SmoothQuant migrates outliers from activations into weights, but compressed-tensors’ W8A8 uses a per-token / per-channel activation-quant granularity that is invariant to exactly that rescaling. The recovery it shows in torchao is real — and specific to torchao’s activation-quant granularity. It is a measurement artifact of the framework you measured in, and it does not exist in the one you serve from.

The lesson: a recovery technique’s benefit is a property of the (technique × quant-granularity × backend) triple, not the technique alone. “SmoothQuant recovers X%” is meaningless without naming where it was measured — and because this one is mechanical (it’s about the serving backend’s activation-quant granularity, not the model), it’s a bit-identical no-op on every architecture we tried. Our re-eval gate flagged it automatically — it re-scored the served model, saw no change, and refused to claim a recovery.

Finding 4 — per-layer mixed precision helps at 1.5B and hurts at 7B

“Keep the fragile layers in higher precision, quantize the rest” is the intuitive recipe for mixed-precision. We tested whether a cheap per-layer fragility ranking (measure each layer’s int4 sensitivity once) predicts which layers, kept at fp16, actually recover the served int4 model.

On Qwen2.5-1.5B it works and transfers cleanly. Keeping 4 layers fp16, choosing which four:

kept fp16 (1.5B) served perplexity
all-int4 12.66
top-4 (by our ranking) 11.28
random-4 11.73
bottom-4 11.96

Monotonic: the ranking is real. So we scaled to 7B, expecting a larger effect (outlier features sharpen with model size). Instead it inverted:

kept fp16 (7B), K=4 served perplexity
all-int4 9.93
top-4 10.10
random-4 10.35
bottom-4 10.16

At 7B, keeping any layers fp16 made the served model worse than plain all-int4 — and the “most fragile” layers were the worst to protect. Why: bigger models are more int4-robust, so int4+GPTQ is already near-lossless at 7B — there is no per-layer fragility left to exploit, because GPTQ’s calibration correction already absorbed it. The cheap ranking measures fragility without the recovery method; the served model has it. And pulling the early layers out of the GPTQ set disrupts its sequential error-compensation, so protecting them backfires.

The lesson: per-layer sensitivity measured on a bare quantizer does not predict a served model that includes a recovery method — and the whole optimization’s value shrinks with scale exactly where you’d want to deploy it. The 1.5B result alone would have justified building a feature that’s worthless at 7B. Measuring at the scale you deploy is the only thing that caught it.

Finding 5 — for recsys, AUC tells you the wrong component to protect

Everything above is LLMs. Recommendation models are the more interesting quantization target precisely because they’re heterogeneous — big and small embedding tables, cross layers, a deep MLP — so a single precision can’t fit all of them, and you have to decide per component. We trained a DCN-v2 on MovieLens-1M and int4-quantized each component in isolation.

By AUC (the offline ranking metric), int4 is nearly free and flat across components — nothing to see. But AUC is rank-based; it is blind to calibration, and a recommendation model’s output is a probability that feeds a downstream auction, where a calibration shift is real money. Measured by calibration error (ECE) instead:

component (int4) ΔAUC ΔECE
head −0.0025 +0.0058
side embeddings −0.0010 +0.0045
cross layers −0.0009 +0.0023
big embeddings −0.0018 +0.0014
deep MLP −0.0001 −0.0004

The calibration ranking is not the AUC ranking: by AUC you’d protect the big embeddings (2nd); by calibration they’re 4th, and the side embeddings jump to 2nd. The deep MLP is free on both. So the offline proxy metric points you at the wrong component to keep in higher precision.

The lesson: measure the metric your deployment actually cares about, per component — not the aggregate offline proxy. (Honest caveat: on a small model the magnitudes are small; this effect wants production-scale tables to become large. But the ranking flip is the point, and it’s already visible.)

What actually works (and ships)

The findings are cautionary; the constructive half is real too. Measuring the served model let us build a loop that ships what it verifies:

These live behind one command — firefly optimize <model> --quality-bar <b> → a servable compressed-tensors checkpoint + a vllm serve line + the measured evidence.

The engine underneath

None of this is quantization-specific. The core is a divergence-attribution engine — capture → compare → attribute — that hooks every layer’s activations and names the first place two model executions diverge, down to the attention head or ATen op. Pointed at serving stacks instead of quantization, the same engine is a numerical-parity CI gate; it’s how we found, in an earlier phase, that FLASH_ATTN vs XFORMERS diverge at exactly one attention head across a 60× model-scale range, and surfaced a live vLLM/FlashInfer bug that silently zeroed two of Qwen-7B’s attention heads. The attribution is the moat: it’s why the tool can say which layer a quant broke and whether the model it ships matches the one it measured — which is the whole story above.

Honest scope