blog: Disaggregated Serving in TensorRT-LLM

[Shixiaowei02]

In this tech blog, we will introduce Disaggregated Serving in TensorRT-LLM, mainly covering the following topics:

• We explain why disaggregated Serving is needed and what problems it solves in LLM inference.
• Architecture and usage of trtllm-serve, Dynamo, and Triton Inference Server. We describe how these components work and how to use them in practice.
• We introduce how KV cache exchange is designed and optimized to improve performance.
• We share performance results, explain how they were measured, and show how to reproduce DeepSeek R1 benchmarks.

By NVIDIA TensorRT-LLM Team

[Shixiaowei02]

@xmchen1987 I can't add you to the review list, so please also review it. Thank you!

[juney-nvidia]

/bot run --comment "No need to run full CI"

[tensorrt-cicd]

PR_Github #9474 Bot args parsing error: usage: /bot [-h]
{run,kill,skip,submit,reviewers,reuse-pipeline,reuse-review} ...
/bot: error: unrecognized arguments: --comment No need to run full CI

[juney-nvidia]

/bot skip --comment "No need to run full CI"

[tensorrt-cicd]

PR_Github #9475 [ skip ] triggered by Bot

[tensorrt-cicd]

PR_Github #9475 [ skip ] completed with state SUCCESS
Skipping testing for commit 012f575

[juney-nvidia]

/bot skip --comment "No need to run full CI"

[tensorrt-cicd]

PR_Github #9490 [ skip ] triggered by Bot

[tensorrt-cicd]

PR_Github #9490 [ skip ] completed with state ABORTED

[juney-nvidia]

/bot skip --comment "No need to run full CI"

[tensorrt-cicd]

PR_Github #9493 [ skip ] triggered by Bot

[tensorrt-cicd]

PR_Github #9493 [ skip ] completed with state SUCCESS
Skipping testing for commit fe8839f