3 months ago
- The transition from frontier model APIs to building your own open-source LLM inference stack often starts due to financial, legal, or engineering concerns.
- The initial decision point is whether to buy from managed inference providers or build with open weights and your own hardware, with this post focusing on the build path.
- Starting with a single replica using serving engines like vLLM, SGLang, or TensorRT-LLM typically works well, handling batching and KV cache management effectively.
- Optimizations within a replica include quantization (e.g., FP8 for +14% throughput) and speculative decoding (like EAGLE), but good defaults are still evolving.
- As concurrency increases, latency can degrade drastically at the 'knee' point, necessitating multiple replicas and KV-cache-aware routing to maintain performance.
- Prefill/decode disaggregation (as in Mooncake) separates compute-bound and memory-bandwidth-bound phases, allowing heterogeneous hardware usage for cost efficiency.
- Model mixes with smaller and larger models require intelligent routing via gateways (e.g., LiteLLM, RouteLLM) to manage costs while maintaining quality for different requests.
- Caching strategies, including prefix caching across storage tiers (HBM to NVMe) and response caching, help reduce recomputation and lower costs as traffic grows.
- Observability across the stack remains challenging, with metrics scattered across components; tools like OpenTelemetry and Langfuse provide pieces but lack end-to-end visibility.
- Key gaps include integrated fleet management, workload-aware tooling, and better defaults to replace folklore, with multimodal serving posing additional challenges.