- 38% of GPTBot rules from the top 10,000 sites were written in Q4 2023, shortly after GPTBot launched and the New York Times sued OpenAI.
- Most sites diligently maintain their robots.txt files but focus on blocking training crawlers like GPTBot, ClaudeBot, and PerplexityBot, while largely ignoring answer-time bots (e.g., OAI-SearchBot, Claude-User).
- Among sites blocking training crawlers, 71% for Anthropic, 53% for OpenAI, and 50% for Perplexity have no rules for answer-time bots, and explicit allowances for answer-time bots are rare (under 4%).
- 144 out of 5,577 sites explicitly allow answer-time bots, showing early adoption of an opt-in approach for AI-generated answers.