Cloudflare: Stay discoverable in search while disallowing AI training
5 hours ago
- Cloudflare introduces a new 'Disallow AI Training' setting that allows site owners to block AI training while remaining indexed in search results.
- Mixed-use crawlers from Apple, Google, and Microsoft honor or have committed to honor this setting, eliminating the previous tradeoff between search discoverability and AI training.
- The 'Block' and 'Block on pages with ads' settings now apply to all training crawlers, including mixed-use crawlers, while 'Disallow AI Training' preserves search access for accountable bots.
- Existing site owner preferences will be migrated automatically, with new options for ad-supported sites to block AI training and agents on pages with ads.
- Accountable crawler operators (Apple, Google, Microsoft, and others) provide opt-out mechanisms for AI training, AI summaries, URL-level transparency, and assurance that blocking training doesn't affect search rankings.
- Cloudflare is working on more granular controls for AI summaries by early next year, allowing site owners to set preferences once on Cloudflare rather than with each operator.
- AI summaries have mixed impacts: they reduce site visits but attract higher-intent traffic, and site owners need control over how much of their content is included.
- Cloudflare aims to evolve standards (e.g., ai-prefs) and collaborate with organizations like the IETF to establish interoperable protocols for crawler controls.