Four bits, then a cliff

The 17 GB Q4_K_M quantization of Qwen3.8 27B scores the same as the full 55 GB BF16 model on Terminal-Bench 2.1, an 89-task agentic coding benchmark. The quantized file fits on a 24 GB card such as an RTX 4090 with room left for about 64k tokens of context. The full-precision model, at 55 GB, is beyond most consumer hardware.

The numbers come from a benchmarking run on Modal GPUs that cost around $3,000. The author ran Qwen3.8 27B at several bit-widths against three benchmarks: GPQA Diamond (graduate-level science), IFBench (instruction following), and Terminal-Bench 2.1 (agentic coding). The point was to answer something KL-divergence and top-1 prediction stats cannot: whether a quantized model actually solves fewer tasks, or just paraphrases the same answer with slightly different tokens. Reddit threads are full of people reporting that every quantization, even 8-bit, feels dumber. A single changed token can be harmless rephrasing or a logic error that ends the output early, and divergence metrics do not tell the two apart.

A few setup details, if you reproduce this: llama.cpp from the 16 August 2026 build (earlier builds do not run this model), Unsloth quants (v2 for the 2-, 4- and 8-bit files, v3 for 1-bit), and an F16 KV-cache kept at full precision regardless of model quantization, costing about 2.3 GB per 32k tokens. Unsloth replaced the v2 files on 19 August 2026, so the exact files used for most tests are no longer available.

Down to four bits, nothing breaks

On GPQA Diamond the author first replicated Qwen’s official BF16 numbers, which is the hard part of any benchmark run. Scores then held flat down to 4-bit, with only the 2-bit quant dipping slightly, inside Wilson 95% confidence intervals that are already conservative for run-to-run noise. The reasoning effort setting moved the score far more than quantization did. The default is xhigh, and the best GPQA results needed around 8k reasoning tokens.

IFBench showed no change down to a 2-bit model under 11 GB, though that configuration left room for only about 4k tokens of context.

Terminal-Bench 2.1 is the result that matters for anyone running a local coding agent. BF16 replicated the stated score and so did Q4_K_M. Q8_0 was skipped, with its value interpolated between 4-bit and full precision. The 2-bit UD-Q2_K_XL fell noticeably but still landed at the level of Opus 4.7 or Gemini 3.1 Pro: below frontier, well short of useless. On tasks both models solved, the 2-bit version used the same number of turns as BF16 and wrote about a quarter more tokens to get there.

The 1-bit cliff

Quantization damage is nonlinear. There is no measurable change, then a small decline, then a collapse, and the collapse lands at 1 bit. On GPQA Diamond the best 1-bit model scores around random-guess level and the smallest one scores below it. More reasoning makes this worse: at xhigh effort the 1-bit scores drop below the low-effort scores, because the model keeps reasoning until the token budget runs out and returns an empty answer.

Unsloth advertises a UD-IQ1_S quant at 6.2 GB (without MTP) that retains about 72% top-1 accuracy while being 89% smaller. On these benchmarks the missing 28% is where the task-solving lives. Another user on r/LocalLLaMA reported the same under the title “Qwen3.8 27b 1bit brain damage quant.”

What to run

Running this was not cheap. Benchmarking through an API is already costly; renting GPUs to do it is more so. The author used Modal with NVIDIA L40S (48 GB, the same Ada Lovelace chip as the RTX 4090 but with twice the memory), H100 (80 GB) and H200 (141 GB), running a few parallel streams rather than Multi-Token Prediction, since the binding constraint is fitting the model plus its KV-caches in memory. For scale, DeepSeek V4 Flash 0731, a 284B model, runs about $0.10 per million output tokens from the cheapest OpenRouter providers. The author does not attribute that gap to any single cause and notes the math changes once the hardware is yours.

The practical rule is to fit the largest quant that leaves room for the context you need. For most work that is Q4_K_M, which showed no measurable penalty on any of the three benchmarks at 17 GB. For simpler tasks UD-Q2_K_XL is fine and buys back memory. Skip 1-bit for anything you expect to reason or solve tasks. Two knobs still bite: the KV-cache here stayed at F16 and cache quantization is reportedly more fragile than weight quantization, so test it before trusting it; and the reasoning effort setting has a real cost, since xhigh can overthink a task into an empty answer at low bit-widths.