Cloudflare Crawl Endpoint

3 days ago

Browser Rendering introduces a new /crawl endpoint in open beta for crawling entire websites with a single API call.
Automatically discovers, renders, and returns pages in HTML, Markdown, and structured JSON formats.
Supports asynchronous crawl jobs with job ID tracking for results.
Key features include multiple output formats, crawl scope controls, automatic page discovery, and incremental crawling.
Offers static mode for faster crawling of static sites and honors robots.txt directives.
Available on both Workers Free and Paid plans.

Hasty Briefsbeta