distributed inference

Run large language models at home, BitTorrent‑style

3 months ago

Run large language models like Llama 3.1, Mixtral, Falcon, or BLOOM at home using a consumer-grade GPU or Google Colab via a BitTorrent-style distributed network.
Achieve inference speeds of up to 6 tokens/sec for Llama 2 (70B) and 4 tokens/sec for Falcon (180B), suitable for chatbots and interactive apps.
Access advanced capabilities beyond basic APIs, including fine-tuning, custom sampling, model path execution, and hidden state inspection, with PyTorch and Transformers flexibility.