{"id":22597,"date":"2026-07-23T01:23:04","date_gmt":"2026-07-23T01:23:04","guid":{"rendered":"https:\/\/scannn.com\/in-house-llm-serving-at-netflix\/"},"modified":"2026-07-23T01:23:04","modified_gmt":"2026-07-23T01:23:04","slug":"in-house-llm-serving-at-netflix","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/in-house-llm-serving-at-netflix\/","title":{"rendered":"In-House LLM Serving at Netflix"},"content":{"rendered":"\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
<\/p>\n
\n
<\/div>\n<\/div>\n

<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n

<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

By AI Platform\u2019s Model Runtime team and Inference team<\/em><\/p>\n

Introduction<\/h2>\n

Most organizations consume LLMs through hosted APIs. Netflix went further \u2014 we run the full stack ourselves, from model deployment through inference, inside our existing production environment rather than a separate ML silo. Some of those decisions weren\u2019t obvious, and a few revealed their trade-offs only under production load.<\/p>\n

This post focuses on the choices where alternatives were seriously considered: engine selection, model packaging, API surface design, deployment strategy, and output constraints enforcement. The goal is to share not just what was built, but why \u2014 and what production revealed that the design phase didn\u2019t anticipate.<\/p>\n

Architecture Overview<\/h2>\n

Member-scale ML at Netflix is fronted by a unified JVM-based serving system that handles the end-to-end flow for downstream consumers: routing and A\/B test logic, candidate generation, feature fetching, inference, post-processing, and logging at each stage. Both real-time and cached batch paths are supported. Figure 1 shows the two ways callers reach inference today: the gRPC path through this serving system and a direct HTTP path used by newer LLM-driven applications.<\/p>\n

Where inference runs depends on the model. Small CPU models run in-process, avoiding remote-call overhead. Larger models need GPUs \u2014 the serving system handles pre- and post-processing locally but delegates inference to a remote service, Model Scoring Service (MSS)<\/strong>. MSS is the shared inference backend, supporting XGBoost, TensorFlow, PyTorch, and LLMs behind a unified interface, with NVIDIA Triton Inference Server underneath managing model loading, batching, and GPU scheduling.<\/p>\n

On top of Triton sits a Java control plane that handles deployment, versioning, health checking, autoscaling, and multi-region rollout. Model authors package their artifacts and configure the deployment; the control plane provisions GPU instances, configures Triton, and orchestrates zero-downtime upgrades.<\/p>\n

\n
Press enter or click to view image in full size<\/span><\/p>\n
\"\"<\/picture><\/div>\n<\/div>
Figure 1. Serving Architecture Overview<\/figcaption><\/figure>\n

Design Decisions and Implementation<\/h2>\n

Four decisions shape this platform \u2014 engine, packaging, API surface, and rollout \u2014 presented in dependency order, since each one constrains the next.<\/p>\n

vLLM as the Paved-Path Engine<\/h3>\n

The platform was originally built on TensorRT-LLM, a performant inference engine at the time and already integrated with Triton \u2014 the compute backend in use within MSS.<\/p>\n

By summer 2025, two things had shifted: open-source engines had largely closed the performance gap with specialized stacks, and our workload mix had broadened to include embedding generation, prefill-only inference for ranking and retrieval, autoregressive decoding, and custom models with non-trivial per-step constraint logic. We re-benchmarked against this mix and selected vLLM as our paved-path engine<\/strong> on operational fit:<\/p>\n