RC RANDOM CHAOS

uv adds file-level dedup to its wheel cache, trimming ~10% of cache size

· via Hacker News

Original source

uv: Deduplicate all files in the wheel cache

Hacker News →

Astral’s uv package manager already deduplicated cached wheels at the archive level—download the same wheel from two sources and they’d share one entry—but never looked inside those wheels. A new preview change stores every extracted file under its BLAKE3 hash in a dedicated bucket and hardlinks the objects back into their original locations, so identical files within and across wheels now occupy disk once. Installation is untouched, and cache cleanup reclaims an object only when its hardlink count falls to one. On the author’s machine this reclaimed about 545 MiB, roughly 10% of the cache, at the cost of a sub-4% penalty on cold installs and no measurable hit on warm ones.

Two follow-ups blunt even that small cost. One reuses a single 64 KiB hashing buffer across an entire wheel instead of allocating and zeroing a fresh buffer per file—cutting PyTorch’s buffer allocations from 11,120 to one and shaving 7–9% off cold installs for NumPy, SymPy, and PyTorch in paired benchmarks. The other rewrites cleanup on macOS to batch hardlink-count lookups via getattrlistbulk, dropping cache-clean and prune times from roughly 386 ms to about 100 ms on a shard of 87,000 objects.

The work landed in uv 0.12.8 (released August 31, 2026) behind the content-addressed-cache preview flag, alongside a related fix that stops concurrent uv processes from downloading and extracting the same remote wheel more than once. It’s a representative slice of Astral’s tooling strategy: aggressive, benchmark-driven optimization of the unglamorous plumbing—cache layout, buffer reuse, syscall batching—that dominates real-world Python install times.

Read the full article

Continue reading at Hacker News →

This is an AI-generated summary. Read the original for the full story.