RC RANDOM CHAOS

Shopify moved inventory reservations from Redis to MySQL—and it held at peak

· via Hacker News

Original source

Shopify replaced Redis with MySQL for inventory reservations–and it scaled

Hacker News →

Shopify’s oversell protection places a short hold on inventory the moment a buyer starts payment, then permanently deducts it once the charge succeeds. Getting this wrong either double-sells the last unit or falsely marks stock as sold out, and at a platform processing $5.1 million in sales per minute at its Black Friday 2025 peak, both failure modes are costly. The system ran on Redis for years, but keeping reservations in Redis while the inventory ledger lived in MySQL meant the reserve and claim steps couldn’t share a single atomic transaction—leaving gaps that could oversell or undersell. Consolidating everything into MySQL let the team wrap both steps in one ACID transaction and delete those failure modes outright.

The design that made MySQL viable hinges on modeling one row per sellable unit rather than a single row with a quantity column, which had buckled under contention in earlier attempts. Reserving units becomes a matter of selecting and moving rows with SELECT … FOR UPDATE SKIP LOCKED, so concurrent checkouts step over each other’s locked rows instead of queuing on one hot row. To keep scans fast, Shopify caps the pool at 1,000 rows per item/location and refills it from the ledger via a replenishment process; if a flash sale drains the pool, a single lock-guarded transaction replenishes inline while others wait, avoiding a thundering herd at the cost of some latency on that request.

The more instructive lessons came from low-level lock behavior rather than the high-level schema. A composite primary key on the columns used in the WHERE clause cut InnoDB from two row locks per reservation to one. Switching those transactions from the default REPEATABLE READ to READ COMMITTED eliminated gap and supremum locks that were blocking replenishment inserts and causing deadlocks. And enforcing a consistent lock-acquisition order across the reserve and claim paths—always deleting from the units table before inserting—broke the circular waits that had produced deadlocks when the two operations touched tables in different orders. The takeaway: at this throughput, primary-key and isolation-level choices are throughput decisions, not footnotes.

Read the full article

Continue reading at Hacker News →

This is an AI-generated summary. Read the original for the full story.