GitHub Actions and Pages Hit by Hours-Long Runner Outage, Now Resolved
GitHub has resolved a significant incident that degraded GitHub Actions and Pages, with the core problem tracing back to runners being handed jobs that were no longer valid. Both GitHub-hosted and self-hosted runners were affected, and Actions Runner Controller (ARC) pods on Kubernetes could become stuck idle, requiring users to delete the pods via kubectl or redeploy ARC so replacements would spawn. At the low point, only 30–40% of queued jobs were succeeding, and webhook processing was throttled to roughly 15%, meaning many pushes and pull requests never triggered workflow runs.
Recovery came in stages as GitHub deployed fixes for the invalid-job assignments and for self-hosted runners that weren’t picking up work. Job success rates climbed back to 97% and then 99% as global queues drained and webhook-triggered workflows returned to full throughput. Adjacent services took collateral damage during the window, including Copilot code review, the Copilot coding agent, and GitHub Pages, while migrations through GitHub Enterprise Importer were deliberately paused as a precaution.
The practical fallout is that some workflow-triggering events were dropped and cannot be replayed automatically, so developers may need to push a new commit, update a PR, or manually re-run affected workflows. GitHub says upcoming releases of Actions Runner and Actions Runner Controller will add an automatic recovery mechanism to eliminate the manual pod cleanup, and a full root cause analysis is promised once available.
Read the full article
Continue reading at Hacker News →This is an AI-generated summary. Read the original for the full story.