We replaced RAG with a virtual filesystem for our AI documentation assistant
a day ago
- #AI-assistant
- #documentation
- #virtual-filesystem
- ChromaFs is a virtual filesystem built to enable assistants to explore documentation like a codebase, using commands like grep, cat, ls, and find.
- Traditional sandbox approaches had high latency (~46 seconds p90 boot time) and significant infrastructure costs (~$70,000/year at scale), making them unsuitable for frontend assistants.
- ChromaFs intercepts UNIX commands and translates them into queries against a Chroma database, reducing boot time to ~100 milliseconds and eliminating marginal compute costs by reusing existing infrastructure.
- It bootstraps a directory tree from a compressed JSON document stored in Chroma, enabling fast local resolution of commands like ls and cd, with access control based on user permissions.
- Pages are reassembled from chunks in Chroma, and lazy file pointers allow on-demand fetching for external resources, while all write operations are blocked to maintain a read-only, stateless system.
- Grep operations are optimized by using Chroma for coarse filtering and Redis caching, then executing fine filtering in-memory, achieving millisecond performance for recursive queries.
- The system is deployed in production, handling hundreds of thousands of users across 30,000+ daily conversations, with instant session creation, zero marginal cost, and built-in RBAC.