Abusive AI Web Crawlers: Get Off My Lawn
2 days ago
- #botnet
- #AI-crawlers
- #web-security
- Significant increase in abusive web crawler traffic reported.
- Automated blocking tools in place for SEO and AI crawlers, brute force attacks, and security scanners.
- Recent attacks involve many IPs with few requests each, using random User-Agents, complicating blocking efforts.
- Shared hosting server averages over 1.5 million fraudulent requests daily from 290,000 unique IPs.
- 5.1 million unique IPs identified, with 3.4 million making only a single request.
- 2.4 million unique User-Agents observed, 1.9 million seen only once.
- Majority of requests from consumer ISPs, with Brazil as the top contributor; UK accounts for 2%.
- Suspected botnet of compromised Android SetTop Boxes leased to an AI crawler to evade blocks.
- Over 22,000 distinct ASNs identified, including 200+ UK-based networks.