gzip as a language model: how a compressor learns to write Shakespeare

There is a deep equivalence between compression and prediction: the number of bits a compressor spends on a symbol is the negative log of the probability its internal model assigns to it. Every compressor therefore contains a prediction model, even one nobody deliberately built. Building on the ‘Language Modeling is Compression’ paper, this experiment asks whether ordinary gzip can generate text. gzip’s DEFLATE algorithm encodes repeated byte sequences within a 32 KiB sliding window as cheap back-references, so a candidate continuation that echoes text already in the window compresses to almost nothing. That gives a scoring function: prime the window with a corpus, then rank possible continuations by the compressed length of context plus candidate — shorter means more expected.

Scoring is easy; generating is harder. Greedy byte-by-byte selection fails because gzip only reports whole-byte lengths, so adding one byte often leaves the compressed size unchanged and ties drown the signal in quantization noise. The fix is a beam search that looks ahead across a span of bytes before committing, keeping the most compressible partial continuations at each step. A further trick: only the last few bytes of generated output are retained in the scoring context, because DEFLATE prices nearby matches cheaply and would otherwise collapse into copying its own recent output verbatim.

Primed on tiny Shakespeare, the tool (‘GziPT’, actually built on zlib) produces text that is not coherent but clearly captures character names, line breaks, and the cadence of the source — more structure than a stock compressor has any obvious right to know. It runs as a single file of standard-library Python with no weights and no training, and the code is on GitHub. The takeaway is less a practical text generator than a vivid demonstration that prediction and compression are two views of the same thing.