The Atlantic's AI bot blocking strategy
13 hours ago
- #AI-crawlers
- #content-protection
- #publishing-strategy
- The Atlantic has implemented a scorecard system to evaluate AI crawlers, blocking those that don't drive traffic or subscriptions.
- Only AI crawlers with licensing deals or those that bring significant value (traffic or subscribers) are allowed access.
- The Atlantic uses Cloudflare's tools to track and block unauthorized AI crawlers, focusing on those that scrape content without permission.
- Publishers face challenges with Google's AI crawler (Google-Extended) as blocking it may affect search traffic due to bundled AI Overviews.
- The Atlantic plans to use Cloudflare's Content Signals Policy to communicate usage preferences to AI crawlers, though compliance isn't guaranteed.
- AI bot traffic has surged, with some bots generating billions of requests monthly without sending any traffic back to publishers.
- Publishers are reevaluating blanket AI bot-blocking strategies, as it may incentivize bots to evade detection.
- The Atlantic's approach aims to maintain leverage for future negotiations or litigation with AI companies.