AI Statistics Rarely Trace Back to Verified Studies
Researchers traced 1,285 statistics cited by five major AI assistants (ChatGPT, Perplexity, Gemini, Claude, Google AI Overviews) after asking them 200 questions requiring numbers. Following citations backward through multiple pages exposed severe sourcing problems: only 40% traced to original studies confirming the number, 8% reached studies where the number or context had changed, 40% hit unchecked sources, and 11% led nowhere. ChatGPT performed best while citing the fewest statistics per answer (1.3 versus Claude’s 6.8), though the other four landed between 38-46%, too close to rank.
Most cited pages contained the correct figure 82% of the time, but only 21% were original research. The rest were company blogs, news stories, and statistics roundups repeating figures from elsewhere. Starting from research firms or government pages, 61% of trails reached matching studies; from company blogs, 37%; from statistics roundups, 35%. Of the statistics that reached original studies, 17% had changed along the way. Agreement among assistants provided no sourcing advantage: four statistics quoted by all five had no original study behind them, and stats cited by multiple assistants showed no better sourcing than others.
Read the full article
Continue reading at Hacker News →This is an AI-generated summary. Read the original for the full story.