New Weight Layout Pushes Ternary LLMs Below the 1.58-Bit Floor

Ternary language models store each weight as -1, 0, or +1, and their memory cost has long been benchmarked against the information-theoretic limit of about 1.585 bits per weight. In practice the popular five-trit-per-byte packing scheme actually lands at 1.625 bits because it assumes all three values are equally likely. The authors challenge that assumption by profiling 29 real ternary models and finding that the distribution is heavily skewed toward zero — in the sparsest model, more than half the weights are zero.

Their scheme, BITCOS, exploits this skew with a two-part layout: a dense bitmap marking which weights are nonzero, plus a packed vector holding just the signs of those nonzero weights. Its cost scales with sparsity at 2 minus the zero density in bits per weight, beating five-trit packing in 26 of the 29 models tested and dropping to 1.485 bits on the sparsest — below the theoretical floor that assumed equiprobable symbols. The format is designed to unpack cheaply, with hand-tuned routines for AVX-512, AVX2, and Intel Xe2 GPUs.

The practical payoff is throughput, not just smaller files. Compared against production-grade ternary matrix-vector kernels at realistic sparsity levels, BITCOS delivers up to a 1.28x speedup, and end-to-end inference across five client and server CPU and GPU platforms improves decode throughput by up to 1.18x on CPUs and 1.27x on GPUs. For anyone deploying low-bit models on commodity hardware, it’s a reminder that treating weight distributions as uniform leaves measurable performance on the table.