distributed computing

Bilingual

Distributing LLM Inference in DwarfStar
6 days ago
- High-end NVIDIA setups require expensive servers and power; alternatives like Apple hardware and DGX Spark exist but have limitations.
- Mac Studio M3 Ultra 512GB can run DeepSeek v4 PRO at usable speeds (150 t/s prefill, 10-13 t/s decoding) for around $12k.
- The future is uncertain: NVIDIA prices are unlikely to drop, and RAM shortages may delay higher-end Apple hardware like M5 Ultra.
- Currently, MacBook M5 Max 128GB offers the best value, running 2-bit quantized models at ~500 t/s prefill and ~35-40 t/s decoding for $6-7k.
more...
React for Two Computers — overreacted
5 days ago
- The article explores the conceptual differences between function calls (verbs/recipes) and tags (nouns/blueprints), showing how tags represent potential computations that are passive and open to interpretation.
- It introduces a framework for splitting computations across time and space, using 'Early' and 'Late' worlds to represent distributed computation across two computers, with tags serving as a bridge between them.
- Key concepts include 'Components' (capitalized functions that embed rather than introspect arguments) and 'Primitives' (lowercase functions that require eager evaluation), along with mechanisms like 'import tag' and 'import rpc' for crossing computer boundaries.
- The article demonstrates how to split a React-like program into Early and Late Components, with interpret() dissolving components outside-in and perform() handling primitives inside-out, enabling client-server architecture without traditional network calls.
more...
A SETI Home for AI-Assisted Research
21 days ago
- SETI@home was a project using home computer idle compute for distributed signal analysis in the Search for Extraterrestrial Intelligence.
- Now, with AI subscriptions and local agent software, unused token allowances could be contributed to crowdsourced scientific or mathematical efforts.
- Small teams and individuals combining domain expertise with AI systems have achieved research successes like solving Erdős problems.
- A key challenge is aligning the frontiers of unknown problems with current AI system capabilities to effectively use increased compute.
more...
Mesh LLM: distributed AI computing on iroh
22 days ago
- Mesh LLM is a distributed AI computing system that pools GPUs and memory across multiple machines, presenting them as a single OpenAI-compatible API.
- It offers three methods for serving requests: running locally on a GPU, routing to a peer with the model loaded, or splitting large models across machines as a pipeline.
- The system uses iroh endpoints for node identity and networking, handling NAT traversal and relay fallback, with two relays maintained for open internet reliability.
- Communication rides on QUIC's ALPN negotiation, with separate protocols for mesh management, control, and split model activations.
more...

distributed computing

Distributing LLM Inference in DwarfStar

React for Two Computers — overreacted

A SETI Home for AI-Assisted Research

Mesh LLM: distributed AI computing on iroh