How Google Sees Your Site
2 days ago
- The article analyzes Google bot traffic using HAProxy logs, focusing on IP addresses, user agents, and JA4 TLS fingerprints to identify known and undocumented Google bots.
- Google admits that some bots ignore robots.txt rules; the author uses ASN lookups and JA4 hashes to distinguish official Google traffic from impostors, especially from GCP.
- Only 0.4% of total site traffic comes from Google's official ASN (15169), while GCP traffic (3.09%) may include customer-run bots, indicating Google bot traffic is well-controlled.
- Numerous Google bots were found, including known ones like Googlebot and GoogleOther, as well as undocumented ones like 'Google' (bare) and 'Chrome Privacy Preserving Prefetch Proxy', some of which may be used for AI training.
- JA4 hashes help group suspicious traffic, such as a vulnerability scanner (abuse.xmco.fr) appearing from Google LLC IPs, and a diverse set of Chrome user agents from a single JA4 fingerprint, suggesting Google tests page behavior across browser versions.
- Blocking Google fully is difficult, but specific JA4 fingerprints (e.g., t13d181300_...) can be targeted to restrict certain bots without losing search engine access.