2 months ago
- The argument that self-hosting LLMs saves money is inaccurate when considering utility and hardware costs, unless prioritizing data security.
- Local inference on high-end GPUs like an RTX 4090 consumes 0.025 kWh for 18k tokens, costing ~$0.0095 in electricity, which is 2.5x to 3.5x higher than centralized API costs such as DeepSeek V4 Flash at ~$0.0038.
- Idle power consumption for local systems adds significant costs: a dual-GPU Nvidia desktop idling at 90W costs $898 over three years, making total ownership 26x higher than API costs for 1M tokens/day.
- Local hardware struggles due to memory bottlenecks during inference, with VRAM limits on GPUs (e.g., 32GB on RTX 5090) and bandwidth restrictions on unified memory (e.g., 546 GB/s on M4 Max), slowing token generation for large models.
- Key developments needed include native ternary quantization in hardware to reduce energy use and smart-sleep runtime management to minimize idle power consumption.