Microsoft called it theft; the crawler maps your attack surface

Microsoft’s own data shows its Copilot “answer engine” cut click-through rates to The New York Times’ domain by as much as 93 percent compared with traditional Bing search. That figure sits in newly unsealed filings from the copyright case the Times brought against OpenAI and Microsoft three years ago, and it is the kind of number that undercuts the defense both companies are running.

The case turns on fair use. Courts have mostly accepted that training a model on copyrighted text can qualify, the way parody, news reporting, or criticism does. One prong of that test asks whether the new use substitutes for the original and harms its market. The unsealed material is full of people inside both companies saying, in writing, that it does.

A January 2024 presentation by Brent Hecht, Microsoft’s director of Applied Science, called the click-through decline a “doom loop” that would “hurt the performance of our models and the entire web at the same time.” Another Microsoft document frames it as a supply problem: “It is highly unusual that an end-product threatens the economic foundations of its essential suppliers, but that is the situation we have created for our LLM business with respect to its ‘content supply chain.’”

OpenAI’s head of ChatGPT, Nick Turley, wrote that publishers face an “existential threat” from products that are “largely substitutive” and “will get more and more substitutive as they get better.” President Greg Brockman called the models “excellent at news.” Under oath, Satya Nadella agreed that talking to a chatbot “has substituted … giving you the information right there on the website on the AI platform versus needing to go to the underlying source.” For a fair-use argument that depends on the output not competing with the input, that is an unhelpful set of internal quotes to have written down.

The filings also lay out how the content got in, and the volumes are large. OpenAI’s mid-training datasets alone hold more than 91,692 copies of works from the Times, the Daily News, and the Center for Investigative Reporting. One Common Crawl-derived dataset held more than two million documents from nytimes.com. A dataset built from something called Project Mango contained at least 160,903 unique works from the news publishers. OpenAI delivered its entire GPT-3 training dataset to Microsoft, which used it to work out how to put OpenAI’s models into its own products. Microsoft sent data back through initiatives called Project Taxi and Project Mango. Content was scraped from the Bing Index. OpenAI built datasets such as WebText and WebText2 that leaned heavily on scraped news, and pulled millions of articles from Common Crawl, the free, open crawl repository.

Two details stand out for anyone who has written a scraper. OpenAI employees allegedly worked out how to get past paywalls without being detected; when researcher Nick Ryder told Brockman about a “hack to get around nytimes paywall,” Brockman replied, “ah nice.” And the filings describe deliberate stripping of copyright notices from training data before it reached the model, on the reasoning that researchers “wouldn’t want model outputting” copyright notices to users. Removing that information is a distinct legal exposure from the copying itself.

The theft language is the companies’ own. In a January 2023 memo, Hecht called the scraping “an astonishing theft of unprecedented proportions” and “the largest theft of labor in human history.” A separate Microsoft document warns of a “real risk” that generative AI could “significantly disrupt the employment of the very people who generated the data on which the foundation model was trained.” Nadella, in a deposition this year, said “anything that is paywalled should be licensed by anyone who wants to use it … for grounding or training,” and said that had he known OpenAI trained on paywalled content, he would have invoked Microsoft’s right to make OpenAI retrain its models.

Read all of it with the caveat the Times itself carries: most of this comes from the Times’ own brief rather than the sealed exhibits, and the lines are quoted without their surrounding context. The Trump administration, for its part, filed a brief defending OpenAI’s unlicensed training. Steven Lieberman, counsel for the Daily News, said the evidence “shows that OpenAI and Microsoft knew that what they were doing was wrong.” OpenAI and Microsoft did not respond to requests for comment.

If you publish on the open web, the ingestion paths named in the filings already hold your content: Common Crawl and the Bing Index. The paywall did not stop ingestion in this account, because it was treated as an obstacle to route around rather than a boundary to respect.