New agents.txt file found on DreamHost

3 hours ago

DreamHost automatically added an agents.txt file to the root of existing websites, similar to default files like robots.txt.
The file contains directives to discourage LLM training, allow retrieval-augmented generation (RAG), and block access to sensitive directories.
The implementation is based on a proposal later renamed to agent-manifest.txt, but DreamHost used outdated syntax and rolled it out retroactively without notice.
While the defaults are sensible for new sites, the unannounced update and incomplete implementation are criticized as problematic.

Hasty Briefsbeta