Cloudflare to introduce pay-per-crawl for AI bots
10 months ago
- #Content Monetization
- #AI
- #Cloudflare
- Publishers and content creators face a binary choice: allow AI crawlers free access or block them entirely.
- Cloudflare introduces a third option: 'Pay per crawl', allowing content creators to charge AI crawlers for access.
- The system uses HTTP response code 402 ('Payment Required') to facilitate transactions between crawlers and content owners.
- Publishers can set a flat rate for access, choose to allow, charge, or block specific crawlers, and negotiate outside the system.
- Crawlers must authenticate using Web Bot Auth proposals to prevent spoofing and include payment intent in headers.
- Two payment flows exist: reactive (discovery-first) and proactive (intent-first), with financial settlements handled by Cloudflare.
- The initiative aims to evolve with dynamic pricing, granular licenses, and support for agentic interactions in the future.
- Currently in private beta, interested parties can sign up or contact Cloudflare for more information.