2 months ago
- Netflix built an in-house LLM serving platform using vLLM and Triton, integrated within existing production infrastructure.
- Key decisions included selecting vLLM as the engine for extensibility and debuggability, and using the vLLM backend in Triton for model packaging to decouple frontend upgrades.
- The system supports an OpenAI-compatible HTTP frontend alongside gRPC for ecosystem compatibility, and employs Red-Black or Versioned deployment strategies for zero-downtime rollouts.
- Operational challenges addressed include boot sequence optimization with model caching, unified metrics endpoints, and scaling constrained decoding with vLLM V1 to overcome CPU bottlenecks.
- Future investments focus on system prompt compression, asynchronous scheduling, vectorized logits processors, and lower-precision models to improve performance and efficiency.