AI Scrapers Now Eat 20% of git.kernel.org's CPU Rendering Commits

Konstantin Ryabitsev of kernel.org reports that AI training crawlers have become a permanent drain on Linux’s public git infrastructure. Across five geo-distributed nodes with 90 CPU cores, roughly 14–16 are perpetually busy doing nothing but rendering git commits as HTML for scrapers — around 20% of total capacity, and far spikier in practice. By his estimate, only about 2% of git.kernel.org’s traffic is now legitimate; the rest is bots harvesting pre-AI, LLM-free kernel history as prized training data. The irony is that all of this content is trivially available via a plain git clone, yet crawlers instead walk billions of individual commit, diff, and patch URLs across the repo’s ~922 near-identical forks — the most wasteful path possible.

The post reads as a chronicle of an escalating and losing arms race. Early defenses were easy: bots identified themselves by user-agent, then were caught by IP, then by banning entire cloud ASNs. But crawlers have shifted to “proxy SDK monetization” — routing through millions of residential and mobile IPs (often embedded in consumer devices like smart TVs), each making a handful of requests and never returning, making IP bans useless. Deploying Anubis, a proof-of-work challenge that forces clients to burn CPU on a SHA-256 puzzle, worked initially and bought months of relief. But bots kept adapting: they now solve difficulty-4 and even difficulty-5 challenges, with 33% of the daily 6M commit requests powering through — while difficulty-5 leaves legitimate mobile users waiting seconds and their phones running hot.

With no clean fix in sight, kernel.org is resorting to degrading its own service: disabling features, gating expensive operations, and cutting the number of crawlable URLs, meaning anonymous users will lose functionality. The broader signal matters beyond one project — it’s a concrete data point on how AI data hunger is externalizing real infrastructure costs onto open, community-run resources, with the maintainers’ only real hope being that the AI bubble deflates or that crawlers eventually learn to just clone the repos.