5 days ago
- High-end NVIDIA setups for local LLM inference are expensive; alternatives include Apple hardware (Mac Studio, MacBook M5 Max) and DGX Spark, each with trade-offs in memory bandwidth and compute.
- Mac Studio M3 Ultra with 512GB can run DeepSeek v4 PRO at ~150 t/s prefill and ~10-13 t/s decoding for ~$12k, while the M5 Max 128GB laptop offers ~500 t/s prefill and ~35-40 t/s decoding for ~$7k, making it the current best deal.
- Future of local inference is uncertain due to NVIDIA pricing and RAM shortages limiting Apple upgrades; distributed inference across multiple machines becomes attractive.
- Two traditional distributed approaches: sequential layer splitting (activations only, good for prefill with micro-batching) and vertical split with RDMA (parallelizing expert execution, viable for models with large routed experts like Pro).