Stop crawling my HTML you dickheads – use the API
a day ago
- #AI-criticism
- #APIs
- #web-scraping
- The author criticizes the tendency to outsource critical thinking in the AI era, preferring brute force over efficient problem-solving.
- Their website is frequently targeted by scrapers despite offering multiple API alternatives (WordPress JSON API, ActivityPub, oEmbed, plain text).
- The author highlights the availability of structured data via APIs and sitemaps, urging scrapers to use these instead of parsing HTML.
- Similar issues occur with the OpenBenches project, where scrapers ignore GeoJSON and APIs, opting for inefficient HTML scraping.
- The author pleads with LLMs and scrapers to stop scraping HTML and use the provided APIs.
- Comments reflect shared frustrations, with suggestions like AI tar-pits and prompt-poisoning to deter scrapers.
- Others note the persistent, unnecessary scraping of static websites despite available APIs.