Tailscale's six-month outage hunt ends at a 16-year-old SQLite corruption bug

Tailscale runs its control plane on internal shards, each backed by a single SQLite database written by one Go process—a textbook single-writer setup that had run cleanly since 2022. Starting in August 2024, a backup-reading pipeline flagged a corrupted database, and integrity checks confirmed it. What looked like a one-off recurred 19 times across six months. Because the control plane holds only tailnet and device metadata (never private keys or traffic), no sensitive data leaked, but every incident forced engineers to halt the affected shard’s control plane while they repaired or restored it. During that window, already-connected devices kept talking, but new or changing devices couldn’t learn the network, and admins lost the console and API.

The bug was maddeningly hard to pin down. There were no relevant recent code changes, no common thread across shards, customers, features, load, or time of day, and no way to reproduce it synthetically. The team resorted to deploying passive forensic telemetry in production to catch the corruption in the act, contracted the SQLite core developers for direct expertise, and methodically ruled out suspects like broken POSIX locks on close(), mishandled SQLite-owned memory, and unsafe multithreaded access. In parallel they hardened operations—hard-stopping shards on detected corruption, continuously running integrity checks over backups, and tightening runbooks—cutting recovery time from over an hour to under one.

A key breakthrough came from a transaction-logging pipeline that streamed every mutating SQL statement to a separate file. Because SQLite’s single-writer, serializable model produces a fully linear, deterministic history (something Postgres or MySQL couldn’t offer), replaying that log against a known-good snapshot offered a low-loss recovery path—and surfaced the clue that ultimately traced the fault to a long-standing SQLite WAL-reset bug roughly 16 years old. The episode is a reminder that ‘boring,’ battle-tested infrastructure can still harbor deep, rare defects that only surface at scale, and that vendor collaboration plus disciplined forensic instrumentation is often the only way to corner them.