RC RANDOM CHAOS

misa77: an LZ codec built for read-heavy workloads, decoding 2x faster than LZ4

· via Hacker News

Original source

Show HN: misa77 - a codec that decodes 2x faster than LZ4 (at better ratios)

Hacker News →

misa77 is a new open-source, LZ-family compression codec aimed squarely at write-once, read-many workloads, where files are compressed once but decoded repeatedly. Its author trades away encoding speed to maximize single-threaded decompression throughput while holding memory use nearly constant — under 5 MB for compression regardless of input size, and effectively zero for decompression. Without an entropy backend it can’t match ratios from tools like zstd, so the honest comparison point is LZ4 at high effort, and against that baseline misa77 claims to sit on the Pareto frontier for decode speed versus ratio across most data shapes, frequently decoding faster even when rivals compress worse.

An interesting property is that misa77 decompresses highly compressible data faster, so spending more effort at compression time yields both a better ratio and quicker decoding — a synergy that motivates the experimental modes shipped in src/experimental/ that try to shape the compressed stream to be friendlier to modern CPU microarchitectures. Benchmarks on the Silesia corpus and enwik8 (run on a pinned Intel performance core via a fork of lzbench) show level 0 beating LZ4’s decode speed on all 12 Silesia files, with the only real losses on near-incompressible data like x-ray, where LZ4 essentially degrades to a memcpy.

The project is early — v0.2.x, MIT-licensed, C++20, little-endian 64-bit only, with runtime AVX2/SSE2 selection on x86-64 and an auto-vectorizable portable path elsewhere. The current decoder assumes valid input and treats malformed streams as undefined behavior; an input-safe decoder is slated for v0.3.0, and the format may still change. It ships as a static library plus a dependency-free misa CLI for single files. Anthropic’s Claude Opus 4.8 and Fable 5 are credited with helping build the tooling and CLI.

Read the full article

Continue reading at Hacker News →

This is an AI-generated summary. Read the original for the full story.