Sites that block AI training crawlers mostly ignore the answer time bots

4 hours ago

38% of GPTBot rules from the top 10,000 sites were written in Q4 2023, shortly after GPTBot launched and the New York Times sued OpenAI.
Most sites diligently maintain their robots.txt files but focus on blocking training crawlers like GPTBot, ClaudeBot, and PerplexityBot, while largely ignoring answer-time bots (e.g., OAI-SearchBot, Claude-User).
Among sites blocking training crawlers, 71% for Anthropic, 53% for OpenAI, and 50% for Perplexity have no rules for answer-time bots, and explicit allowances for answer-time bots are rare (under 4%).
144 out of 5,577 sites explicitly allow answer-time bots, showing early adoption of an opt-in approach for AI-generated answers.
Many sites (883 out of 7,627) block access to robots.txt, preventing AI bots from reading the rules, which may indicate anti-bot stances or security measures.
The study highlights a blind spot: sites focus on the 'training war' but neglect to decide on policies for answer-time fetches, which are becoming monetizable through toll systems like Cloudflare's.
Effective AI integration requires not just allowing answer-time bots but ensuring web pages can provide clear, answerable content for AI queries.

Hasty Briefsbeta