Use DNSBL to block AI crawlers in Caddy
9 months ago
- #DNSBL
- #Caddy
- #Security
- Wiki was taken down again due to bot scanning through proxies.
- Caddy Defender Plugin alone was insufficient for this attack.
- Considered using DNSBL (DNS Blackhole List) for defense.
- Checked IP addresses against DNSBLs; SpamRATS' RATS-Spam (spam.spamrats.com) was suitable.
- Found and tested caddy-matcher-dnsbl, a Caddy plugin for DNSBL filtering.
- Successfully implemented DNSBL filtering, improving defense.
- Documentation update: Multiple DNSBLs can be used by specifying providers in the configuration.
- 403 status code can be replaced with other 4xx codes for logging blocked requests.
- Use 'sudo tcpdump -i any -n port 53' to monitor DNSBL queries.
- Some DNSBLs block public DNS servers; running a local Unbound DNS server can help.