Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

3 hours ago

94.8% of audited websites are never named in AI assistant answers to buying-intent questions.
Only 1.9% of 5,978 assistant answers across ChatGPT, Claude, Gemini, and Perplexity mentioned the business being audited.
Gemini had the highest mention rate at 2.9%, while 8.9% of sites block at least one AI crawler, mainly training crawlers.
54.6% of sites publish structured data, but only 19.3% use LocalBusiness schema for name, address, and category.
The study separates answer presence (outcome) from machine readability (input), with crawl signals read directly from websites.
Data comes from 458 domains audited between March and August 2026, with cross-assistant queries starting July 23, 2026.
Self-selected sample skews toward small and mid-sized businesses actively managing their web presence.

ChatGPT采用多种数据管道：serp用于开放网络，labrador用于授权出版商，bright用于Bright Data爬虫，oxylabs用于Oxylabs爬虫。
某些查询（如'文本'类型）跳过网络搜索并使用训练数据，这影响了SEO机会。
'扇出查询'显示ChatGPT针对复杂任务运行多个子查询，包括特定站点的探测。

在排名前10,000的网站中，38%针对GPTBot的规则是在2023年第四季度编写的，当时正值GPTBot刚推出不久，且《纽约时报》起诉OpenAI的时期。
大多数网站会勤勉地维护其robots.txt文件，但主要专注于屏蔽GPTBot、ClaudeBot和PerplexityBot等训练爬虫，而基本忽略了即时回答型机器人（例如OAI-SearchBot、Claude-User）。
在屏蔽训练爬虫的网站中，71%针对Anthropic、53%针对OpenAI、50%针对Perplexity的网站没有为即时回答型机器人设置规则，而明确允许即时回答型机器人的情况很少（不到4%）。

GitHub AI activity surged: new repositories up 79.2%, pull requests up 138.8%, issues up 146.8%.
Jobs in call centres dropped 7.3%, newspaper publishers down 6.3%, and AI-exposed industries now represent 0.8% of all jobs.
AI-related package downloads skyrocketed: openai on npm up 428.2%, on PyPI up 472.3%; anthropic on PyPI up 704.8%.

ChatGPT采用多种数据管道：serp用于开放网络，labrador用于授权出版商，bright用于Bright Data爬虫，oxylabs用于Oxylabs爬虫。
某些查询（如'文本'类型）跳过网络搜索并使用训练数据，这影响了SEO机会。
'扇出查询'显示ChatGPT针对复杂任务运行多个子查询，包括特定站点的探测。

在排名前10,000的网站中，38%针对GPTBot的规则是在2023年第四季度编写的，当时正值GPTBot刚推出不久，且《纽约时报》起诉OpenAI的时期。
大多数网站会勤勉地维护其robots.txt文件，但主要专注于屏蔽GPTBot、ClaudeBot和PerplexityBot等训练爬虫，而基本忽略了即时回答型机器人（例如OAI-SearchBot、Claude-User）。
在屏蔽训练爬虫的网站中，71%针对Anthropic、53%针对OpenAI、50%针对Perplexity的网站没有为即时回答型机器人设置规则，而明确允许即时回答型机器人的情况很少（不到4%）。

GitHub AI activity surged: new repositories up 79.2%, pull requests up 138.8%, issues up 146.8%.
Jobs in call centres dropped 7.3%, newspaper publishers down 6.3%, and AI-exposed industries now represent 0.8% of all jobs.
AI-related package downloads skyrocketed: openai on npm up 428.2%, on PyPI up 472.3%; anthropic on PyPI up 704.8%.

Hasty Briefsbeta