- Cloudflare announced the first Content Independence Day a year ago to help website owners block AI bots and launched a Pay-Per-Crawl marketplace.
- The conversation around AI bots has evolved from simply blocking training to seeking nuanced solutions, recognizing that website owners need more options than just blocking all automation.
- Small websites face a dilemma: allow AI training for discoverability or block it and risk being unfindable, which unfairly advantages large search providers.
- AI is now integrated into many services, like Google's answer engine, shifting search from AI-sorted results to direct answers on results pages.
- Cloudflare's updated approach classifies bots based on behavior—Search, Agent, and Training—rather than just labeling them as AI, to provide better transparency and control.
- New management options allow website owners to control AI traffic based on these three use cases, with defaults set to block Training and Agent bots on ad pages while allowing Search.
- Multi-purpose crawlers like Googlebot will be handled according to all their behaviors, and customers can opt out of default configurations before September 15, 2026.
- BotBase is launched as a visibility tool for Enterprise customers, tracking all known bots with classifications and enabling detailed management and filtering.
- A new 'content use' signal is introduced for robots.txt, allowing website owners to specify how bots can use their content—immediate, reference, or full—enhancing control over content reuse.
- The definition of 'Verified' bots is updated to align with category-based allowances, and the verification process is made more transparent and accessible to bot operators.
- Cloudflare experiments with transitive trust using the Forwarded header to maintain trust across intermediaries, ensuring that trusted operators retain access even through layers.
- These changes aim to give website owners more control over content usage, promote transparency, and adapt to the evolving web ecosystem, with all new options available now.