Agentic Web Index flags bots faking AI-crawler identities like ClaudeBot

Known Agents’ Agentic Web Index tracks how automated traffic — AI scrapers, real-time fetchers, search crawlers, and autonomous browsing agents — moves across a network of more than 5,000 participating websites. Alongside metrics on bot-versus-human traffic and robots.txt compliance, it maintains a spoofing and security section that is the basis for the claim in the headline: it counts visits that assert a recognized agent’s identity but fail that agent’s supported authentication, such as verified source IPs or Web Bot Auth message signatures. When a request claims to be a known crawler yet can’t prove it, the index treats it as impersonation and ranks the most-spoofed identities and the paths those requests hit most often.

The methodology is deliberately cautious about what this proves. A failed authentication check flags likely impersonation of a named agent, but it does not identify the actual software or operator behind the request — so ‘someone spoofing ClaudeBot’ means traffic wearing that label, not confirmation of who sent it or why. Agents that publish no verification method are excluded entirely, and only verified traffic is attributed to those that do. Rates are computed per site and then averaged so a few high-traffic domains can’t dominate.

The practical takeaway for defenders is that a well-known AI crawler’s user-agent string is trivially forgeable and worthless as an allow-list criterion on its own. Cryptographic or IP-based bot authentication is what separates a legitimate ClaudeBot fetch from a scanner hiding behind its reputation — and the index’s own data suggests that gap is being actively exploited. The figures are framed as directional trends across an observed, self-selected network rather than a precise census of the web.