Why Compression and LLMs Are Secretly the Same Problem

An ngrok engineering explainer makes the case that data compressors and large language models are, at their core, doing the same job: predicting what comes next. The piece walks through compression fundamentals to build the argument. It first distinguishes minification—which merely discards syntax machines don’t need—from ‘true’ compression, which exploits redundancy. Run-length encoding serves as the entry point, collapsing repeated characters into character-count pairs to shrink a 224-bit string to 96 bits.

Modern compressors like gzip and Brotli are then broken into three cooperating parts: transforms that preprocess data (sometimes adding redundancy to enable later gains), models that assign probabilities to each symbol, and entropy coders that emit the final bitstream. The article zooms in on arithmetic coding, showing how an entire string can be encoded as a single binary fraction by repeatedly narrowing a [0,1) range according to symbol probabilities—compressing a 56-bit ASCII string into just 10 bits, then reversing the process to decode it losslessly from that one number.

The payoff is the model itself. Because an entropy coder’s efficiency depends entirely on the quality of the probabilities it’s fed, better prediction yields better compression—and predicting the next symbol from context is precisely what a language model does. That equivalence reframes LLMs as extraordinarily powerful statistical models whose predictive accuracy is, in an information-theoretic sense, a measure of how well they compress. It’s a useful mental model for anyone trying to understand what modern AI systems are actually optimizing.