2 days ago
- High-end NVIDIA setups require expensive servers and power; alternatives like Apple hardware and DGX Spark exist but have limitations.
- Mac Studio M3 Ultra 512GB can run DeepSeek v4 PRO at usable speeds (150 t/s prefill, 10-13 t/s decoding) for around $12k.
- The future is uncertain: NVIDIA prices are unlikely to drop, and RAM shortages may delay higher-end Apple hardware like M5 Ultra.
- Currently, MacBook M5 Max 128GB offers the best value, running 2-bit quantized models at ~500 t/s prefill and ~35-40 t/s decoding for $6-7k.
- Distributed inference is promising: traditional layer splitting and expert parallelism across machines, but communication speed is a bottleneck.
- A novel third approach is model ensemble, where different models run independently and combine logits or select best continuations, showing improved performance.
- The author expresses interest in further exploring these distributed techniques for local inference.