The Free and Open Web Is Under Attack at the IETF
6 hours ago
- #IETF standards
- #internet freedom
- #web scraping
- Automated access tools like crawling and scraping are essential for a free internet, enabling journalism, research, security analysis, and archiving.
- Publishers and Big Tech are threatening open internet access by seeking to restrict bots that use public web content for AI, citing economic concerns and infrastructure strain.
- Some proposals in the IETF aim to give websites the power to block legitimate scraping via preference signals in robots.txt, potentially making these signals legally binding.
- Efforts like the Web Bot Auth group could allow sites to cryptographically identify and block bots, enabling restrictions based on payment, which would harm researchers, archivists, and startups.
- These changes would undermine the benefits of the open web, affecting uses such as investigations, archiving, accessibility tools, and government accountability research.
- EFF and allies are actively opposing these IETF proposals to protect the right to freely access the internet using automated tools and maintain neutral internet standards.