4-bit Qwen3.8 27B matches the full model on real tasks; 1-bit falls apart

A researcher spent roughly $3,000 renting Modal GPUs to test how aggressively Qwen3.8 27B can be quantized before it loses real-world capability. Rather than relying on token-prediction proxies like KL-divergence, which reveal statistical drift but not whether a model still solves problems, the study ran actual task benchmarks: GPQA Diamond for graduate-level science, IFBench for instruction-following, and the 89-task Terminal-Bench 2.1 for agentic coding. The BF16 baseline reproduced Qwen’s officially reported numbers, giving the downstream comparisons credibility.

The headline finding is that quantization damage is nonlinear. Down to 4-bit, there is essentially no measurable difference from the full 55 GB model. The 17 GB Q4_K_M build matched BF16 on Terminal-Bench 2.1 and fits on a 24 GB card such as an RTX 4090 with room for about 64k tokens of context. Even 2-bit remained usable — it broke only modestly on coding, still landing near Opus 4.7 or Gemini 3.1 Pro, though it burned about a quarter more tokens to reach the same answers. At 1-bit the model collapses to random-guess accuracy on GPQA Diamond, and higher reasoning effort actually makes it worse because it exhausts its token budget and returns empty answers. Unsloth’s claim that its 1-bit quants retain ~72% top-1 accuracy is technically true but misleading, since the missing 28% is where task-solving lives.

The practical takeaway: fit the largest model plus context you can into available GPU memory, and treat quantization as a tool rather than a threat. Q4_K_M is a safe default for most work, and Q2 suffices for simpler tasks — directly rebutting the common complaint on forums that any quantization makes local models ‘dumber.’ The report also flags that reasoning-effort settings (xhigh by default) swing scores dramatically and can cause overthinking, and notes KV-cache quantization as a likely follow-up, since it may degrade quality more readily than weight quantization.