Streaming AI Agent Desktops with Gaming Protocols
17 days ago
- #Moonlight Protocol
- #Desktop Streaming
- #AI Agents
- Helix built sandboxes for AI agents with interactive desktop environments streamed to browsers using Moonlight, a game streaming protocol.
- Moonlight was designed for single-player gaming, but Helix needed multi-user access to shared AI agent sessions.
- Initial workaround: 'Apps mode' where Helix API pretends to be a client to start agent sessions, but it's fundamentally single-user.
- Solution: 'Lobbies mode' in Wolf (Moonlight server) designed for multiplayer gaming, allowing multiple users to connect to the same agent session.
- Current challenges with lobbies mode include input scaling issues, video corruption on some clients, and less dynamic resolution flexibility.
- Architecture involves Helix API, Moonlight-web, Wolf in Kubernetes, and desktop containers with GPU-accelerated Wayland.
- Moonlight offers low latency, hardware encoding, network resilience, and multi-platform support, but required adaptation for AI agent use.
- Key lessons: Protocol assumptions can create challenges, workarounds add complexity, and open-source collaboration is valuable.
- Future plans include fixing lobbies mode bugs, enabling multi-user support, and improving mobile client compatibility.