Residential proxies turn millions of hijacked devices into a scraping arms race

LWN reports that the scraping onslaught against websites now arrives as coordinated bursts from millions of unique residential and mobile IP addresses, each touching a site only two or three times with faked user-agent strings meant to pass as ordinary browsers. The traffic is powered by “residential proxies”: software running on ordinary people’s devices — often without their knowledge — that takes orders from central command-and-control nodes, fetches pages, and relays the results. Operators range from purely criminal botnets built on malware-compromised systems (Google’s takedown of the IPIDEA network earlier this year briefly cut LWN’s scraper load) to nominally legitimate firms like Bright Data that offer “free” VPNs and paid SDKs in exchange for routing traffic through users’ devices. Compromised media-streaming boxes have become a major carrier of this software.

The deeper worry is that whoever controls these networks can run arbitrary code touching whatever networks those millions of devices sit on — scraping is only the visible use. It remains unclear who pays for the attacks; the well-known frontier AI companies scrape openly, identify themselves, and mostly honor robots.txt, so they aren’t the worst offenders. LWN speculates the real demand comes from a long tail of undisclosed models being trained by companies, governments, and criminal organizations all racing for training data, with the open Internet caught in the crossfire.

Defending against this imposes a heavy tax on everyone: proof-of-work gates like Anubis, CAPTCHAs, login walls, paywalls, and data-poisoning tools such as iocaine. LWN says it recently weathered its largest attack yet without most readers noticing, but declined to detail its defenses since it’s an arms race. Notably, it avoided Anubis — proof-of-work is a weak deterrent when attackers have millions of other people’s machines to compute on — and resisted allowlisting dominant search engines, which would only further entrench an existing monopoly.