In-House LLM Serving at Netflix

2 days ago

Netflix built an in-house LLM serving platform using vLLM and Triton, integrated within existing production infrastructure.
Key decisions included selecting vLLM as the engine for extensibility and debuggability, and using the vLLM backend in Triton for model packaging to decouple frontend upgrades.
The system supports an OpenAI-compatible HTTP frontend alongside gRPC for ecosystem compatibility, and employs Red-Black or Versioned deployment strategies for zero-downtime rollouts.
Operational challenges addressed include boot sequence optimization with model caching, unified metrics endpoints, and scaling constrained decoding with vLLM V1 to overcome CPU bottlenecks.
Future investments focus on system prompt compression, asynchronous scheduling, vectorized logits processors, and lower-precision models to improve performance and efficiency.

Hasty Briefsbeta