Four bits, then a cliff

A 27B-parameter model dropped to 4-bit weights runs on a single 24GB consumer GPU and lands within a point or two of its full-precision self on most tasks. Push the same model to 1-bit and it stops being a model. It produces grammatical noise, loses instruction-following, and fails structured output almost entirely. The distance between those two outcomes is not a gentle slope you can tune your way down. It is a cliff, and on Qwen3.8 27B the edge sits somewhere below 3 bits per weight.

So the production answer is narrow and boring, which is exactly why it works: 4-bit is the default. The K-quant, AWQ, and GPTQ 4-bit families take a model that needs roughly 54GB in FP16 down to about 14-16GB, which is the difference between a rented A100 and a card you already own. In exchange you give up something in the range of one to two points on aggregate benchmarks like MMLU and a small, mostly invisible bump in perplexity. For most inference workloads that trade is free money. 8-bit exists if you have the memory and want to erase even that small gap, but it rarely justifies the extra ~11GB. Everything at 2-bit and below is research territory. It is interesting, it is not shippable, and it should not touch a user.

The real takeaway from a quantization benchmark is not ‘how low can I go.’ It is ‘what is the lowest bit-width that still passes the validation for my specific task.’ Those are different questions with different answers. A summarization pipeline tolerates degradation that a code generator or a JSON-emitting agent will not. The benchmark tells you where the model breaks in general. Your own eval set tells you where it breaks for you, and the second number is the only one you should be making deployment decisions on.

Underneath the numbers, quantization is a compression problem with a hard information floor. A model stores its learned behavior in weight matrices originally represented as 16-bit floats. Quantization maps those floats onto a much smaller set of integer values plus a scale factor, so instead of storing every weight at full precision you store a rounded index and a shared multiplier for a group of weights. At 4-bit each weight gets 16 possible values; at 2-bit it gets 4; at 1-bit it gets 2. The scale factors and grouping are what make modern 4-bit schemes hold up. Methods like GPTQ and AWQ do not round naively. They look at which weights actually matter for the model’s outputs, using calibration data to protect the salient channels and absorb rounding error into the ones that matter less. That is why a good 4-bit quant beats a lazy 4-bit quant by a wide margin even though both use the same bit budget.

The collapse at 1-bit is not a tuning failure, it is arithmetic. With two possible values per weight there is no room to preserve the distribution of magnitudes that the network learned. Every weight becomes essentially a sign, and the fine structure that separates ‘confident’ from ‘slightly confident’ from ‘wrong’ disappears. Worse, error does not stay local. A transformer is a deep stack, and quantization error at one layer feeds forward as corrupted input to the next. At 4-bit that per-layer error is small enough that later layers can still work with what they receive. At 1-bit the error compounds across dozens of layers until the residual stream no longer carries usable signal. This is why the degradation curve is non-linear: you lose almost nothing from 16-bit to 4-bit, a little more from 4-bit to 3-bit, and then the floor drops out. The model does not get gradually dumber. It works, works, works, and then it is gibberish.

This also explains why bigger models quantize better than small ones. A 27B model has enough redundancy across its parameters that aggressive rounding removes slack rather than signal. The same 4-bit scheme applied to a 3B model does real damage, because there is less redundancy to spend. Bit-width is not an absolute quality setting. It interacts with model size, with the quantization method, and with how sensitive your task is to the specific capabilities that low-bit rounding erodes first - long-context reasoning, multi-step instruction following, and exact formatting tend to break before basic fluency does.

The most common mistake is treating quantization as a single linear dial where 8-bit is ‘half as good’ as 16-bit and 4-bit is ‘half again.’ That mental model is wrong in both directions. The high end is nearly flat - you pay almost nothing from 16 down to 4 - and the low end is a cliff, not a ramp. Teams that assume linearity either waste money running 8-bit when 4-bit would have been fine, or they get burned reaching for 2-bit because ‘it’s only a bit less than 3’ and discover their agent can no longer emit valid JSON.

The second mistake is trusting a single benchmark number. A model that scores well on MMLU can still fall apart on your task, because aggregate academic benchmarks reward broad knowledge and say almost nothing about instruction adherence, tool-call formatting, or behavior at long context - the things production pipelines actually depend on. Quantization degradation is task-dependent, and it hits structured, precise, multi-step behavior hardest. If your only evidence is a leaderboard delta, you are measuring the wrong failure mode. The teams that get this right run their own eval set at every candidate bit-width and look at the failures, not just the score.

The third mistake is conflating the different things people call ‘quantization.’ Weight quantization, KV-cache quantization, and activation quantization are separate decisions with separate risk profiles, and they compound. A model with clean 4-bit weights can still degrade badly if you also crush the KV cache to 2-bit to save memory on long context - and when it breaks, people blame the weights. Beyond that, the quantized file you download is only as good as the calibration data and method behind it. Two files both labeled ‘Q4’ can differ by several points depending on whether they were produced with proper calibration or a naive round-to-nearest pass. ‘It’s 4-bit’ is not a spec. The method, the group size, and the calibration set are the spec, and skipping that detail is how a quant that looked fine in a demo quietly underperforms in production.

Start every decision at 4-bit and pick the family by your serving stack, not by benchmark chatter. If you serve on GPU through vLLM or TGI, AWQ or GPTQ are the defaults because they run as native kernels and were built to protect the salient channels during calibration. If you run llama.cpp or a mixed CPU-GPU setup, the K-quants are the right family, and Q4_K_M is the sane starting point. Group size 128 is the standard trade between accuracy and overhead; smaller groups recover a little quality at the cost of more scale factors and memory. None of this should be decided on vibes. Decide the serving stack first, because it constrains which quant families are even available to you, then let your own numbers pick the winner inside that family.

The eval set is the gate, and you build it before you pick a bit-width. Pull 100 to 300 real requests from your actual traffic, or realistic proxies if you cannot, and attach a known-good output to each one. Run the model in FP16 first to establish a ceiling, then run every candidate quant against the same set. Score the thing your pipeline actually depends on: valid-JSON rate, tool-call correctness, field-level accuracy, instruction adherence at your real context length. Do not score MMLU. Then read the failures, not just the aggregate. A quant that loses two points of average accuracy but drops your valid-JSON rate from 99% to 80% has not lost two points of anything you care about - it has broken your agent while looking fine on a leaderboard.

Separate the three quantization decisions and set a fallback ladder before you ship. Weight quantization, KV-cache quantization, and activation quantization are independent knobs with independent risk, and they compound in ways that get misattributed. Keep the KV cache at fp8 or int8 for long context; dropping it to 2-bit to save memory is the single most common way a clean 4-bit model starts emitting garbage on long inputs. Verify provenance on any file you download: the method, the group size, and the calibration set are the spec, and two files both stamped ‘Q4’ can differ by several points. If a candidate fails the gate, step up one bit-width or switch method - do not patch it with prompt tricks that hide the failure until production. And regardless of bit-width, wrap every structured-output step in schema validation with a bounded retry. The model is probabilistic; the system around it should catch the tail.

A concrete version of this: a team wants to serve Qwen3.8 27B on a single RTX 4090 with 24GB for a document-extraction agent that emits JSON. FP16 needs roughly 54GB and does not fit, so quantization is not optional here - it is the reason the deployment exists at all. They assemble an eval of 250 real extraction requests, each paired with the expected JSON, and run it first on a rented A100 in FP16 to get a ceiling: 99.2% valid JSON and high field accuracy. That number is the bar every quant has to clear.

They then test three 4-bit candidates on the 4090. AWQ 4-bit lands at 98.4% valid JSON, occupies about 14.5GB, and leaves headroom for context. GPTQ 4-bit and Q4_K_M come in within a fraction of a point. All three clear the bar, so the weight decision is effectively free - they pick AWQ for kernel speed on vLLM and move on. This is the boring, correct outcome Phase 1 promised: at 4-bit, on a model this size, you pay almost nothing.

The interesting part is what happens next. To handle larger documents they push context to 32k, the KV cache blows past available memory, and someone quantizes the KV cache to 2-bit to make it fit. Valid JSON collapses to 71% - truncated fields, broken nesting, dropped keys. The instinct is to blame the 4-bit weights and go hunting for a ‘better’ quant, which is exactly the misattribution Phase 1 warned about. But because the eval isolates variables, they can see the weights never changed; only the KV cache did. They revert the KV cache to fp8, valid JSON returns to 98%, and it still fits at 32k. They ship AWQ 4-bit weights, an fp8 KV cache, and a schema validator with a single retry. Had they trusted the demo instead of the eval, the 2-bit KV reach would have shipped an agent that looked flawless on short inputs and failed silently on the long documents it was built for.

4-bit is the production floor for a 27B model. Above it you pay almost nothing, and below 3 bits per weight you fall off a cliff, not down a ramp. That means the two decisions most teams agonize over - 8-bit for safety, 2-bit for savings - are usually the wrong decisions to spend energy on. The real work is choosing which 4-bit method fits your stack and proving it holds on your task.

Your eval set is the deployment gate, not the leaderboard. Aggregate benchmarks measure broad knowledge and say almost nothing about the structured, precise, multi-step behavior your pipeline runs on, which is exactly what low-bit rounding degrades first. The lowest shippable bit-width is the one that still passes your task’s validation, and only your own numbers know what that is. If your evidence is a benchmark delta, you are measuring a failure mode you do not have.

And ‘4-bit’ is not a spec. The method, the group size, the calibration set, and which tensors you quantized are the spec - weights, KV cache, and activations are separate risk decisions that compound. Set them deliberately, keep a validation layer around every non-deterministic step, and keep a fallback ladder for when a quant fails the gate. Do that and quantization stops being a gamble and becomes what it should be: a predictable lever that puts a 27B model on hardware you already own, without putting a broken one in front of a user.