GitHub's 8-hour August 17 outage traced to capacity that couldn't keep up with growth
GitHub’s second major August incident took down github.com, authentication, Actions, APIs, pull requests, issues, and Copilot for nearly eight hours on August 17. The trigger wasn’t a bad deploy or config change—it was raw capacity. As traffic hit a new peak, a critical infrastructure component in GitHub’s Central US data center failed to scale, and the resulting pressure cascaded into authentication failures across the platform. Recovery dragged on for Copilot in particular, where a client-side retry loop piled additional load onto systems that were still trying to come back, forcing engineers to tame the retry storm before restoring traffic.
The underlying story is demand outrunning infrastructure: monthly commits have roughly doubled since April, from 1.4 billion to 2.9 billion. GitHub says it has thrown hardware at the problem—over 3 million CPU cores, 120 petabytes of high-speed storage, and more network capacity—while leaning hard on its Azure migration, which now carries about 58% of platform load and half of all Git operations, up from just 12% in May. A forthcoming architecture aims to scale read capacity linearly with the number of readers, starting with the largest monorepos.
The more telling admissions are operational. GitHub concedes its practices didn’t keep pace with the rate of change, and the two August incidents prompted immediate fixes: consistent retry limits, retry budgets, and variable timeouts across service-to-service calls to prevent cascading retry storms, plus a review of low-priority CPU and memory alerts that mask components likely to buckle under sudden spikes. Longer-term work centers on isolating critical systems and stripping out shared dependencies so a single component’s failure can’t take down the whole platform.
Read the full article
Continue reading at Hacker News →This is an AI-generated summary. Read the original for the full story.