Microsoft called it theft; the crawler maps your attack surface

Common Crawl, one of the datasets that trained the current generation of large language models, holds petabytes of text pulled from billions of web pages. Almost none of the people who wrote that text agreed to it. That gap, between what people published and what they consented to, is what a Microsoft executive pointed at when the phrase “the largest theft of labor in human history” started moving around tech circles. Strip the outrage out and you get a plainer question, one worth answering carefully: what actually moves when a model scrapes the web, and what does that movement expose?

What a scraper actually takes off your servers

A scraper is a program that requests pages the same way your browser does, then keeps the parts it wants. It reads your HTML, follows your links, and copies your text into a dataset. The companies building models run named crawlers you can identify in your server logs: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended (Google’s AI training fetch), Bytespider (ByteDance), PerplexityBot. Each one hits your site, downloads what it can reach, and leaves.

The important detail for anyone thinking about security: a scraper touches the same surface an attacker’s reconnaissance tool touches. It enumerates your pages, maps your directory structure, and pulls anything a logged-out visitor can see. If your staff directory, your internal wiki left open, or an old backup folder is reachable without a login, a crawler will find it and a copy will land in someone else’s dataset. The crawler is not trying to breach you. It does not need to. You published the door.

Public and consented are two different states

Most of the legal and moral fight lives in this distinction. A photo on your public profile, a blog post, a Stack Overflow answer, a court record with your name in it: all of it is reachable. Reachable is not the same as offered up for commercial reuse, and the people who wrote it never got asked.

For security purposes the distinction matters because it changes where your data ends up and who controls it after that. Once your text sits inside a training corpus, you cannot delete it. There is no takedown request that reaches into a trained model’s weights. If you posted your home address in a 2014 forum thread and later scrubbed it, the scrubbed version is what visitors see. The 2014 version may already sit in a dataset that gets reused for years. Your delete button controls the live web. It does not control the copies.

Where the actual security risk lives

Three mechanisms turn scraped data into a privacy problem, and none of them require a hacker.

The first is memorization and regurgitation. Large models sometimes reproduce their training data close to verbatim. Researchers at Google, Berkeley, and other groups demonstrated in 2023 that you can prompt production models into emitting exact strings from their training set, including names, phone numbers, and email addresses. If your personal data went into training, a stranger with the right prompt can sometimes pull it back out. The New York Times attached near-verbatim article reproductions to its lawsuit against OpenAI to make this point. The same mechanism that reproduces a paywalled article can reproduce a leaked spreadsheet that once sat on the open web.

The second is membership inference. Even when a model does not spit out your data word for word, an attacker can often determine whether a specific record was in the training set by studying how confidently the model handles it. For an ordinary person that sounds abstract. For a data set of, say, patients from a specific clinic that briefly published records by mistake, membership inference can confirm someone was a patient. The fact of inclusion is itself the sensitive information.

The third is aggregation. Scrapers do not respect the boundaries you kept in your head. You put your employer on LinkedIn, your neighborhood in a running app, your kid’s school in a PTA post, your birthday in a Facebook comment. Each is harmless alone. A model trained on all of it, or a scraping operation feeding a people-search product, stitches the fragments into a profile you never assembled and never approved. The 2021 incidents where scraped data from 533 million Facebook accounts and 700 million LinkedIn profiles showed up for sale were built this way, from public fields, no breach of the login required.

The load nobody puts in the pitch deck

Scraping is not free for the site being scraped. It costs bandwidth, CPU, and money, and the bill lands on the operator.

Automated traffic now makes up close to half of all web traffic, and a rising share of it is AI crawlers. Read the Docs, which hosts open-source documentation, reported that a single crawler pulled 73 terabytes in one month and that blocking AI bots cut their bandwidth bill by thousands of dollars. iFixit and other sites reported crawlers hitting them millions of times in a day. For a small site, an aggressive crawler is close to a denial-of-service event you did not schedule, and it can knock a hobby project offline the same way a real attack would.

This matters for your threat model because it collapses a clean line. The traffic hammering your origin might be a training crawler, a search bot, a competitor scraping prices, or the reconnaissance phase of an attack. They look similar in the logs. Every defense you build against one shapes your exposure to the others.

Scraping sits next to your other attack surface

Treat a scraper as an unauthenticated user with patience and bandwidth, because that is what it is. The same techniques that stop bulk data harvesting also blunt credential stuffing, price scraping, and inventory-hoarding bots. The reverse holds too: a public API that leaks one user’s data per call becomes a full-database dump when a bot calls it a few million times. Several of the largest “scraping breaches” were exactly this, a working feature abused at machine scale.

The defensive lesson is to stop treating scraping as a content or copyright issue and start treating it as a rate and access issue. Ask what a determined anonymous client can pull from you in 24 hours, then decide whether that volume is acceptable. That single question surfaces most of the same weaknesses a penetration test would.

Poisoning runs in both directions

Scraping created a new attack against the models themselves. If you can predict what a crawler will collect, you can plant text designed to corrupt the training run. Researchers showed you can buy expired domains that old datasets still point at and refill them with whatever you want, or edit pages a crawler is about to fetch. A study of web-scale datasets found that poisoning a fraction of a percent of the data is enough to bias a model. Scraped data is data of unknown provenance, and unknown provenance is a supply-chain risk whether it lands in your code or in a model’s weights.

Defenders can use the same asymmetry. Tools like Nightshade and Glaze let artists alter images so that a model trained on them learns the wrong thing, and server-side tarpits like Nepenthes and Cloudflare’s AI Labyrinth feed crawlers endless generated pages to waste their budget. These are not clean solutions. They are a reminder that the scraped web is now contested ground rather than a passive library.

What a site operator can actually do

Start with robots.txt. It is a request, not a wall, but the major AI crawlers do honor it. Disallow the ones you do not want by name: GPTBot, CCBot, Google-Extended, ClaudeBot, Bytespider. It costs a few lines and blocks the compliant majority.

For the crawlers that ignore the rules, you need enforcement, not etiquette. Put a web application firewall or a service like Cloudflare in front of the origin and turn on its bot controls, which now include a one-click block for known AI crawlers. Rate-limit by IP and by request pattern so no anonymous client can pull your whole catalog in a session. Move anything sensitive behind a login, because everything reachable without one should be assumed to be in a dataset already. Audit what a logged-out visitor can see, since that inventory is your real exposure. Do not rely on llms.txt as protection; it is a preferences file crawlers may read, not a control that stops anyone.

What an individual can do, and where it stops

Assume that anything you have ever posted publicly is already training data or soon will be. That framing is more useful than any single setting. Lock down the accounts you can, request removal from the data-broker and people-search sites that aggregate scraped profiles, and stop posting the specific fields that fuel aggregation: your exact location, your employer combined with your schedule, images of documents. Artists and photographers can run Glaze on work before posting it.

The honest limit is that none of this reaches backward. The copies already made are made. What you control is the flow from here forward, and how much new material you hand over each week. The Microsoft executive’s phrase was aimed at labor and ownership, but the security reading is simpler and colder: your public data has already been copied, you cannot recall it, and the only variable you still hold is how much more you keep publishing into a pipe you do not own.

See also: NordVPN for tunneled traffic when operating outside controlled networks.

---

#ad Contains an affiliate link.