web scraping

Bilingual

Judge Rejects Google's Attempt to DMCA Its Way Out of Being Scraped
2 months ago
- A judge dismissed Google's DMCA 1201 lawsuit against SerpAPI for scraping its search results, ruling that the law only protects copyrighted content, not general access control.
- Google's SearchGuard (a CAPTCHA system) was deemed not a valid technological protection measure under DMCA because it controls access to non-copyrighted compilations of public data.
- The court highlighted that Google's own business was built on scraping the web, making its lawsuit appear hypocritical.
- Google can refile a narrower case targeting only copyrighted elements like the Knowledge Panel, but the core DMCA claims were rejected.
- The ruling reinforces open web principles, preventing companies from using DMCA to block scraping of non-copyrighted public data.
Most Googlebots Are Fake
2 months ago
- Complaints about Googlebot overwhelming websites are often misguided because the traffic comes from fake bots impersonating Googlebot.
- The user agent header is a voluntary self-identification that can be easily spoofed by anyone.
- Fake Googlebots are frequently used to bypass bot controls and can be traced to hosting providers like Virtual Machine Solutions LLC.
- To verify a real Googlebot, use reverse DNS lookup or check against Google's published IP ranges.
- Real Googlebot is generally well-behaved, while fake bots are responsible for disruptive traffic.
Hacking my Keyboard to do Less
2 months ago
- The website uses Anubis, a Proof-of-Work system similar to Hashcash, to protect against AI bots scraping content.
- At individual scale, the additional load is negligible, but for mass scrapers it makes scraping more expensive.
- Anubis is a placeholder solution while better fingerprinting methods (e.g., font rendering) are developed to identify headless browsers.
- Users are asked to disable plugins like JShelter that block the modern JavaScript features Anubis requires.
Google Lawsuit Against SerpApi over Scraping Search Results Has Been Dismissed
2 months ago
- Google sued SerpApi for scraping, but a court dismissed most claims under the DMCA.
- The judge ruled that plain search results (URLs, snippets, factual data) are not copyrighted works.
- Google cannot use copyright law to block scraping of uncopyrighted data like aggregated search results.
- For third-party content like knowledge panels, Google failed to show authorization from copyright holders.
- The court allowed Google 21 days to refile if it can prove copyright owner authorization.
- Circumvention techniques (spoofing, rotating IPs, CAPTCHAs) are illegal only if protecting copyrighted works with permission.
- The ruling is seen as a win for scrapers, though not monetarily material for Google.

web scraping

Judge Rejects Google's Attempt to DMCA Its Way Out of Being Scraped

Most Googlebots Are Fake

Hacking my Keyboard to do Less

Google Lawsuit Against SerpApi over Scraping Search Results Has Been Dismissed