Mesh LLM: distributed AI computing on iroh
3 hours ago
- #AI inference
- #peer-to-peer networking
- #distributed computing
- Mesh LLM is a distributed AI computing system that pools GPUs and memory across multiple machines, presenting them as a single OpenAI-compatible API.
- It offers three methods for serving requests: running locally on a GPU, routing to a peer with the model loaded, or splitting large models across machines as a pipeline.
- The system uses iroh endpoints for node identity and networking, handling NAT traversal and relay fallback, with two relays maintained for open internet reliability.
- Communication rides on QUIC's ALPN negotiation, with separate protocols for mesh management, control, and split model activations.
- Inside the main connection, various stream types (e.g., gossip, HTTP tunnels, route queries) are demuxed by a leading byte, enabling efficient peer-to-peer operations.
- Users can install lightweight software (~18 MB) to join the public mesh or set up private deployments, accessible via localhost:9337/v1 for standard OpenAI clients.