From 0d3379cec245371389164b93a944a9125d458a6b Mon Sep 17 00:00:00 2001 From: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> Date: Wed, 13 Aug 2025 06:22:08 +0000 Subject: [PATCH] only keep markdown changes Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> --- benchmarks/cpp/README.md | 4 +- docs/source/advanced/disaggregated-service.md | 11 - ...5_Disaggregated_Serving_in_TensorRT-LLM.md | 2 +- examples/cpp/executor/README.md | 4 +- examples/disaggregated/README.md | 195 ++++++++++++++---- .../slurm/{ => benchmark}/README.md | 0 .../slurm/{ => benchmark}/disaggr_torch.slurm | 0 .../slurm/{ => benchmark}/gen_yaml.py | 0 .../slurm/{ => benchmark}/run_benchmark.sh | 0 .../slurm/{ => benchmark}/start_server.sh | 0 .../slurm/{ => benchmark}/start_worker.sh | 0 .../slurm/{ => benchmark}/submit.sh | 0 .../ctx_extra-llm-api-config.yaml | 6 + .../slurm/simple_example/disagg_config.yaml | 12 ++ .../gen_extra-llm-api-config.yaml | 3 + .../slurm/simple_example/launch.slurm | 36 ++++ examples/wide_ep/slurm_scripts/README.md | 6 +- examples/wide_ep/slurm_scripts/submit.sh | 2 +- 18 files changed, 224 insertions(+), 57 deletions(-) rename examples/disaggregated/slurm/{ => benchmark}/README.md (100%) rename examples/disaggregated/slurm/{ => benchmark}/disaggr_torch.slurm (100%) rename examples/disaggregated/slurm/{ => benchmark}/gen_yaml.py (100%) rename examples/disaggregated/slurm/{ => benchmark}/run_benchmark.sh (100%) rename examples/disaggregated/slurm/{ => benchmark}/start_server.sh (100%) rename examples/disaggregated/slurm/{ => benchmark}/start_worker.sh (100%) rename examples/disaggregated/slurm/{ => benchmark}/submit.sh (100%) create mode 100644 examples/disaggregated/slurm/simple_example/ctx_extra-llm-api-config.yaml create mode 100644 examples/disaggregated/slurm/simple_example/disagg_config.yaml create mode 100644 examples/disaggregated/slurm/simple_example/gen_extra-llm-api-config.yaml create mode 100644 examples/disaggregated/slurm/simple_example/launch.slurm diff --git a/benchmarks/cpp/README.md b/benchmarks/cpp/README.md index 0b89bae6029..ae3287faf06 100644 --- a/benchmarks/cpp/README.md +++ b/benchmarks/cpp/README.md @@ -336,7 +336,7 @@ cd cpp/build `disaggServerBenchmark` only supports `decoder-only` models. Here is the basic usage: ``` -export TRTLLM_USE_MPI_KVCACHE=1 +export TRTLLM_USE_UCX_KVCACHE=1 mpirun -n ${proc} benchmarks/disaggServerBenchmark --context_engine_dirs ${context_engine_0},${context_engine_1}...,${context_engine_{m-1}} \ --generation_engine_dirs ${generation_engine_0},${generation_engine_1}...,${generation_engine_{n-1}} --dataset ${dataset_path} ``` @@ -344,7 +344,7 @@ This command will launch m context engines and n generation engines. You need to for example: ``` -export TRTLLM_USE_MPI_KVCACHE=1 +export TRTLLM_USE_UCX_KVCACHE=1 mpirun -n 7 benchmarks/disaggServerBenchmark --context_engine_dirs ${llama_7b_tp2_pp1_dir},${llama_7b_tp1_pp1_dir} --generation_engine_dirs ${llama_7b_tp1_pp1_dir},${llama_7b_tp2_pp1_dir} --dataset ${dataset_path} # need 6 gpus and 7 processes to launch the benchmark. diff --git a/docs/source/advanced/disaggregated-service.md b/docs/source/advanced/disaggregated-service.md index d8e376d62cb..a9955b940a4 100644 --- a/docs/source/advanced/disaggregated-service.md +++ b/docs/source/advanced/disaggregated-service.md @@ -66,17 +66,6 @@ A. Yes, it's recommended that different executor use different GPUs . We support ### Debugging FAQs -*Q. How to handle error `Disaggregated serving is not enabled, please check the configuration?`* - -A. please set `backendType` of `CacheTransceiverConfig`. -```cpp -ExecutorConfig executorConfig{...}; - -executorConfig.setCacheTransceiverConfig(texec::CacheTransceiverConfig(BackendType::DEFAULT)); -``` - -When the environment variable `TRTLLM_USE_MPI_KVCACHE=1` is set, TRT-LLM will transfer the KV cache using `CUDA-aware MPI`. All executor processes involved must share the same MPI world communicator. Consequently, with `TRTLLM_USE_MPI_KVCACHE=1`, TRT-LLM only supports launching multiple executors via `MPI`. Additionally, the `CommunicationMode` for the executors must be set to `kLEADER` or `kORCHESTRATOR` with `SpawnProcesses=false` for the `disaggregated-service`. These restrictions do not apply when `TRTLLM_USE_UCX_KVCACHE=1` is set. - *Q. Does TRT-LLM support using GPU direct RDMA for inter-node KV Cache transfer?* A. Yes, TRT-LLM supports using GPU direct RDMA for inter-node KV cache transfer. diff --git a/docs/source/blogs/tech_blog/blog5_Disaggregated_Serving_in_TensorRT-LLM.md b/docs/source/blogs/tech_blog/blog5_Disaggregated_Serving_in_TensorRT-LLM.md index 8c9e935585e..8d7682c482f 100644 --- a/docs/source/blogs/tech_blog/blog5_Disaggregated_Serving_in_TensorRT-LLM.md +++ b/docs/source/blogs/tech_blog/blog5_Disaggregated_Serving_in_TensorRT-LLM.md @@ -277,7 +277,7 @@ We also conducted performance evaluations of Qwen 3 on GB200 GPUs. The data indi ### Reproducing Steps -We provide a set of scripts to reproduce the performance data presented in this paper. Please refer to the usage instructions described in [this document](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/disaggregated/slurm). +We provide a set of scripts to reproduce the performance data presented in this paper. Please refer to the usage instructions described in [this document](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/disaggregated/slurm/benchmark). ## Future Work diff --git a/examples/cpp/executor/README.md b/examples/cpp/executor/README.md index fdb5b0d434e..4cc9b72ad98 100644 --- a/examples/cpp/executor/README.md +++ b/examples/cpp/executor/README.md @@ -124,10 +124,10 @@ From the `examples/cpp/executor/build` folder, you can also run the `executorExa ``` ./executorExampleDisaggregated -h ``` -Note setting `TRTLLM_USE_MPI_KVCACHE=1` is required to run disaggregated executor. +Note setting `TRTLLM_USE_UCX_KVCACHE=1` is required to run disaggregated executor. For example, you can run : ``` -export TRTLLM_USE_MPI_KVCACHE=1 +export TRTLLM_USE_UCX_KVCACHE=1 mpirun -n --allow-run-as-root --oversubscribe ./executorExampleDisaggregated --context_engine_dir --context_rank_size --generation_engine_dir --generation_rank_size --input_tokens ../inputTokens.csv diff --git a/examples/disaggregated/README.md b/examples/disaggregated/README.md index 713e69e6be2..196113d9872 100644 --- a/examples/disaggregated/README.md +++ b/examples/disaggregated/README.md @@ -1,38 +1,64 @@ # Disaggregated Serving -To run TensorRT-LLM in disaggregated mode, you must first launch context (prefill) and generation (decode) servers using `trtllm-serve`. +The execution method of disaggregated serving relies on the `trtllm-serve` command. Specifically, compared to the standard usage of `trtllm-serve`, serving requires running this command multiple times to separately start the router and workers (including context and generation) serving components. This document focuses on this approach and provides a detailed guide on how to use it. -## Launching disaggregated servers locally on single node +Please note that disaggregated serving is currently an experimental feature, so the usage described in this document may change in the future. -We use the `cache_transceiver_config` configuration to set up disaggregated serving, which includes the following parameters: +## Startup Procedure + +### Configuration File + +The `trtllm-serve` command supports the `extra-llm-config.yaml` parameter. In the extra LLM configuration file, the `cache_transceiver_config` field is specifically used for disaggregated service. It is mainly used to specify additional parameters required for the KV cache transmission process. ```yaml cache_transceiver_config: + # KV cache transmission backend. Valid options include `DEFAULT` (i.e., UCX), `UCX`, `NIXL`. backend: + # KV cache buffer size. Set it ≥ the maximum ISL (Input Sequence Length) for best performance. max_tokens_in_buffer: ``` -`backend` specifies the communication backend for transferring the kvCache, valid options include `DEFAULT`,`UCX`, `NIXL`, and `MPI`, the default backend is UCX. +The following is an example, consisting of the `ctx_extra-llm-api-config.yaml` and `gen_extra-llm-api-config.yaml` files needed in the sections below. -`max_tokens_in_buffer` defines the buffer size for kvCache transfers, it is recommended to set this value greater than or equal to the maximum ISL (Input Sequence Length) of all requests for optimal performance. +```yaml +# ctx_extra-llm-api-config.yaml -You can use multiple `trtllm-serve` commands to launch the context and generation servers that will be used -for disaggregated serving. For example, you could launch two context servers and one generation servers as follows: +# The overlap scheduler for context servers is currently disabled, as it is +# not yet supported in disaggregated context server architectures. +disable_overlap_scheduler: True +cache_transceiver_config: + backend: UCX + max_tokens_in_buffer: 2048 +``` -```bash -# Generate context_extra-llm-api-config.yml -# Overlap scheduler for context servers are disabled because it's not supported for disaggregated context servers yet -echo -e "disable_overlap_scheduler: True\ncache_transceiver_config:\n backend: UCX\n max_tokens_in_buffer: 2048" > context_extra-llm-api-config.yml +```yaml +# gen_extra-llm-api-config.yaml -# Start context servers -CUDA_VISIBLE_DEVICES=0 trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 --host localhost --port 8001 --extra_llm_api_options ./context_extra-llm-api-config.yml &> log_ctx_0 & -CUDA_VISIBLE_DEVICES=1 trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 --host localhost --port 8002 --extra_llm_api_options ./context_extra-llm-api-config.yml &> log_ctx_1 & +cache_transceiver_config: + backend: UCX + max_tokens_in_buffer: 2048 +``` + +### Basic Usage -# Generate gen_extra-llm-api-config.yml -echo -e "cache_transceiver_config:\n backend: UCX\n max_tokens_in_buffer: 2048" > gen_extra-llm-api-config.yml +For non-SLURM clusters - particularly in single-node, multi-GPU setups, it is recommended to use standard mode. In such cases, the system does not enforce limits on process creation or termination. -# Start generation servers -CUDA_VISIBLE_DEVICES=2 trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 --host localhost --port 8003 --extra_llm_api_options ./gen_extra-llm-api-config.yml &> log_gen_0 & +Suppose we have three CUDA devices on the same machine. The first two devices are used to launch one context model each, and the third device is used to launch one generation model. In this case, the following commands need to be executed. + +```bash +# Start context servers +CUDA_VISIBLE_DEVICES=0 trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 \ + --host localhost --port 8001 \ + --extra_llm_api_options ./ctx_extra-llm-api-config.yaml &> log_ctx_0 & + +CUDA_VISIBLE_DEVICES=1 trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 \ + --host localhost --port 8002 \ + --extra_llm_api_options ./ctx_extra-llm-api-config.yaml &> log_ctx_1 & + +# Start generation server +CUDA_VISIBLE_DEVICES=2 trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 \ + --host localhost --port 8003 \ + --extra_llm_api_options ./gen_extra-llm-api-config.yaml &> log_gen_0 & ``` Once the context and generation servers are launched, you can launch the disaggregated @@ -40,11 +66,16 @@ server, which will accept requests from clients and do the orchestration between and generation servers. The disaggregated server can be launched with: ```bash +# Start proxy trtllm-serve disaggregated -c disagg_config.yaml ``` + where `disagg_config.yaml` contains information about the context and generation servers. For the current example, it would look like: + ```yaml +# disagg_config.yaml + hostname: localhost port: 8000 backend: pytorch @@ -61,13 +92,11 @@ generation_servers: Clients can then send requests to the disaggregated server at `localhost:8000`, which is an OpenAI API compatible endpoint. -## Launching disaggregated servers on SLURM clusters - -Refer to [Disaggregated Inference Benchmark Scripts](./slurm/). -## Sending requests to the disaggregated server +#### Sending requests to the disaggregated server Once the context, generation and disaggregated servers are launched, you can send requests to the disaggregated server using curl: + ```bash curl http://localhost:8000/v1/completions \ -H "Content-Type: application/json" \ @@ -78,33 +107,124 @@ curl http://localhost:8000/v1/completions \ "temperature": 0 }' -w "\n" ``` + Or using the provided client parsing the prompts from a file and sending request to the disaggregated server specified in the `disagg_config.yaml` file at the `chat` endpoint: + ``` python3 ./clients/disagg_client.py -c disagg_config.yaml -p ./clients/prompts.json -e chat ``` +### Launching disaggregated servers on SLURM clusters + +To simplify usage, TensorRT-LLM internally relies on MPI spawning processes. However, some clusters do not offer such process flexibility. In these cases, we provide the `trtllm-llmapi-launch` tool to launch all processes at once. Therefore, when using TensorRT-LLM on a Slurm cluster, please refer to the following method. + +#### Single-Node Execution + +After starting the node and entering interactive mode, you can run the following command to prevent process spawning. + +```bash +# Start context servers +CUDA_VISIBLE_DEVICES=0 trtllm-llmapi-launch trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 \ + --host localhost --port 8001 \ + --extra_llm_api_options ./ctx_extra-llm-api-config.yaml &> log_ctx_0 & + +CUDA_VISIBLE_DEVICES=1 trtllm-llmapi-launch trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 \ + --host localhost --port 8002 \ + --extra_llm_api_options ./ctx_extra-llm-api-config.yaml &> log_ctx_1 & + +# Start generation server +CUDA_VISIBLE_DEVICES=2 trtllm-llmapi-launch trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0 \ + --host localhost --port 8003 \ + --extra_llm_api_options ./gen_extra-llm-api-config.yaml &> log_gen_0 & + +# Start proxy +trtllm-llmapi-launch trtllm-serve disaggregated -c disagg_config.yaml +``` + +#### Multi-Node Execution + +If the model you are running cannot fit within a single node and requires multiple nodes, +we introduce the startup method using [srun](https://slurm.schedmd.com/srun.html) to run parallel jobs. + +```bash +srun -A -p -t