You Don't Need Anubis
6 months ago
- #web-security
- #javascript
- #bot-protection
- Scrapers by LLM training companies have become more aggressive, ignoring robots.txt and using spoofed User Agents and IPs.
- Anubis, a proof-of-work bot protection solution, is often used but may not be necessary for those only needing protection against LLM scrapers.
- Anubis's effectiveness against bots is limited because many LLM scrapers do not run JavaScript.
- A simple 12-line Caddyfile solution can effectively block scrapers by requiring JavaScript to set a cookie, without the drawbacks of Anubis.
- Cloudflare remains the most reliable solution for bot protection, though it has its own drawbacks.
- Anubis is recommended only for specific cases like DDoS protection, not for general use against LLM scrapers.