We auto-convert HTML to Markdown for AI agents
3 months ago
- #AI
- #Cloudflare
- #Markdown
- The way content and businesses are discovered online is shifting from traditional search engines to AI crawlers and agents.
- Markdown is becoming the preferred format for AI systems due to its explicit structure and reduced token usage compared to HTML.
- Cloudflare now supports real-time HTML to markdown conversion for AI agents through content negotiation headers.
- AI agents can request markdown versions of web pages by including 'Accept: text/markdown' in their HTTP headers.
- Cloudflare's feature includes an 'x-markdown-tokens' header to help AI systems manage context windows and chunking strategies.
- Content Signals framework allows content creators to specify how their content can be used by AI, including for training and search.
- Cloudflare has enabled markdown conversion for its Developer Documentation and Blog, inviting AI crawlers to use this feature.
- Alternative methods for markdown conversion include Workers AI and a /markdown REST API for dynamic content.
- Cloudflare Radar now tracks content type insights for AI bot and crawler traffic, including markdown requests.
- Markdown for Agents is available in Beta for Pro, Business, and Enterprise plans, as well as SSL for SaaS customers.