22 days ago
- Mesh LLM is a distributed AI computing system that pools GPUs and memory across multiple machines, presenting them as a single OpenAI-compatible API.
- It offers three methods for serving requests: running locally on a GPU, routing to a peer with the model loaded, or splitting large models across machines as a pipeline.
- The system uses iroh endpoints for node identity and networking, handling NAT traversal and relay fallback, with two relays maintained for open internet reliability.
- Communication rides on QUIC's ALPN negotiation, with separate protocols for mesh management, control, and split model activations.