Nothing to see here (it was Cloudflare crawler hints)

9 months ago

ChatGPT agent is a browser automation feature replacing Operator research preview, set for deprecation on August 31st.
ChatGPT agent's HTTP headers reveal its identity through a Mozilla/5.0 user-agent, but it runs on Linux, indicated by Sec-Ch-Ua-Platform: 'Linux'.
The Signature-Agent header and HTTP Message Signatures standard (RFC 9421) provide a reliable way to identify ChatGPT agent traffic.
Initial belief that ChatGPT was leaking URLs to Bingbot and Yandex was incorrect; it was due to Cloudflare's Crawler Hints feature.
Cloudflare Crawler Hints automatically notifies search engines like Bing and Yandex about new content, explaining the bot visits.
The investigation involved verifying bot identities using tools like Microsoft's Verify Bingbot and reverse DNS lookups for Yandex.

Hasty Briefsbeta