Why 'just schedule it smarter' breaks down when you don't own the control loop

System design has a habit of hiding complexity behind problems that look trivial. The author walks through job queues — systems for submitting, scheduling, and running batch work where throughput matters more than latency (CI pipelines, data-analysis rollups) — and shows how quickly naive reasoning falls apart. He frames the discussion around three lenses picked up over his career: treating queues warily because they tend to sit near-full or near-empty and behave unintuitively around latency, enforcing explicit limits and budgets (à la TigerBeetle’s Tiger Style), and reasoning carefully about fault models, meaning your assumptions about how dependencies fail.

The concrete case is a background job that repacks ‘reference repos’ — aggressively compressed git repositories stored in object storage so large repos can be provisioned by downloading a baseline and fetching a small delta rather than cloning. Two repacking strategies trade off against each other: wholesale repacking recomputes everything from scratch (roughly 7 hours) and yields a repo 50–60% smaller, while incremental repacking reuses existing packing metadata (about 2 hours) and keeps repos more current but larger. Because consumers are far busier on weekdays, the tempting plan is to run cheap incremental packs on weekdays and expensive wholesale packs on weekends.

The catch is that the team doesn’t write the scheduling control loop — the job queue does, and it only exposes a couple of knobs, here a scheduling interval and a concurrency limit of one. The ‘obvious’ fix (shorten the interval to 3 hours and trust the concurrency limit to prevent the 7-hour weekend job from overlapping) sounds airtight but doesn’t actually hold. The broader lesson: when you consume a queue instead of owning its loop, you can only reason about behavior through the limited configuration surface someone else designed, and timer-plus-concurrency semantics have edge cases that defeat back-of-the-envelope scheduling.