Deepseek R1 and V3, FP4 quant, output quality issues at batch size > 2

[pankajroark]

System Info

$> uname -a
Linux deepseek-r1-tensorrtllmworker 5.15.0-1073-gke #79-Ubuntu SMP Thu Jan 16 00:48:08 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

$> nvidia-smi
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.06             Driver Version: 570.124.06     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|

Who can help?

@laikhtewari

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Run trtllm serve

trtllm-serve serve --tp_size 8 --ep_size 4 --max_num_tokens 8192 --max_batch_size 32 --backend pytorch --extra_llm_api_options /node-storage/dsingal/options.yaml /node-storage/nvidia/DeepSeek-R1-FP4

Checkpoint from here: https://huggingface.co/nvidia/DeepSeek-R1-FP4

Config

pytorch_backend_config:
  enable_overlap_scheduler: false
  cuda_graph_padding_enabled: false
  use_cuda_graph: false
    enable_chunked_prefill: true
  #speculative_config:
  #  decoding_type: MTP
  #  num_nextn_predict_layers: 3
kv_cache_config:
  free_gpu_memory_fraction: 0.6
  event_buffer_max_size: 1024
enable_block_reuse: true

To test:

pip install baseten_benchmarks --no-deps

baseten_benchmark --backend generic \
      --api_url http://deepseek-r1/v1/chat/completions \
      --api_key null \
      --model deepseek-ai/DeepSeek-R1 \
      --num_prompts 32 \
      --concurrency 16 \
      --random_input 5000 \
      --output_len 500 \
      --input_type custom \
      --stream \
      --tokenizer deepseek-ai/DeepSeek-R1 \
      --output_file latency.csv \
      --warmup_requests 0 \
      --prompt_style messages

Expected behavior

Reasonable output, adhering to the instruction following pattern.

actual behavior

Some output is good but others are truncated, repetitive or don't sounds like instruction following.

Generated Text: <｜end▁of▁sentence｜>
Generated Text: the<｜begin▁of▁sentence｜># How many moles<｜end▁of▁sentence｜>
Generated Text: Okay
Generated Text: í```python
Generated Text: 21<｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜><｜begin▁of▁sentence｜>
Generated Text: the to analyze this text. Let me start by first understand’
Generated Text: Okay
Generated Text: I thethe,
Generated Text: ``` Frankenstein’s
Generated Text: Okay<｜begin▁of▁sentence｜># How monster
Generated Text: </think>ííííííí of gas are in the container?CO2CO_2
Generated Text: ** I need to know the trial to useReducer with typescript and react
Generated Text: Okay's dive into this passage from Frankenstein. So Frankenstein Volume III, Chapters 21 the context. This is from Mary Shelley's Frankenstein, specifically the part where Victor Frankenstein is recounting his story to Walton. The key elements here are Victor's pursuit of the creature, the perilous journey on the ice, the interaction with Walton's crew, and Victor's eventual death.
Generated Text: Okay
Generated Text: Okay consolation ,,000ortun.kk to and 22. So, first, I need to understand what's happening here. In Chapter 21, Victor is dealing with the aftermath of Clerval's death. He's been in prison, gets acquitted because he was on the Orkney Islands when the murder happened. His father comes to his aid, and they head back to Geneva. Victor is consumed by guilt and depression, thinking about the monster's threat to be with him on his wedding night. He's tormented by the deaths he feels responsible for—William, Justine, Henry.
Generated Text: Okay, I need to take a step back and figure out how to proceed. The user provided a lengthy narrative, and I need to ensure I understand the core of the story and the user's request. The narrative appears to be a continuation or a variation of Mary Shelley's "Frankenstein," focusing on the guilt and torment of Victor Frankenstein after his creation leads to the deaths of William and Justine. The user's request seems to be to continue the story in the same vein, possibly extending the narrative of Victor's guilt and the consequences of his actions.

Comparable output at batch size 1

Generated Text: Okay, so I need to analyze this excerpt from Mary Shelley's "Frankenstein." Let me start by recalling the context. The main character, Victor Frankenstein, is dealing with immense guilt and despair after his creation, the monster, has killed several people close to him, including his friend Henry Clerval. In this part, Victor is recovering from his illness and facing the aftermath of the trial where he was accused of murder but later acquitted. He's now planning to marry Elizabeth, but he's haunted by the monster's threat to be with him on his wedding night.
Generated Text: Okay, let's tackle this. The user provided a lengthy excerpt from what looks like Mary Shelley's "Frankenstein." The task is to analyze it, probably focusing on themes, characters, or literary elements. First, I need to recall the context. This part is near the end of the novel, where Victor Frankenstein is recounting his pursuit of the creature to Captain Walton. Then Walton continues the narrative after Victor's death.
Generated Text: Okay, so I'm trying to understand this passage from Frankenstein. Let me start by summarizing what's happening here. Victor Frankenstein is being confronted by the creature he created, who is demanding a female companion. The creature argues that if Victor creates a mate for him, they will leave humanity alone and live in isolation. Victor is conflicted because he fears the consequences of creating another monster but also feels some compassion. Eventually, he agrees but is filled with dread and postpones the task. Later, Victor's father suggests he marry Elizabeth, but Victor is tormented by his secret promise to the creature. He decides to travel to England to work on the female creature, accompanied by his friend Clerval. The journey is described with vivid scenery, contrasting Clerval's joy with Victor's inner turmoil.
Generated Text: Okay, so I need to figure out what the user is asking here. They provided a lengthy excerpt from what looks like chapters 15 and 16 of Mary Shelley's "Frankenstein." Then they included a partial sentence that breaks off: "had lived too short a time". The user's request is labeled as "think," which I assume means they want me to analyze or continue the thought based on the text provided.
Generated Text: Okay, so I need to analyze this passage from Mary Shelley's "Frankenstein." Let me start by recalling the context. This is part of the creature's narrative where he describes his early experiences after being created. He's learning about the world, observing the De Lacey family, and starting to understand human emotions and language.
Generated Text: Okay, let's tackle this. The user provided a lengthy excerpt from what seems to be Chapters 11, 12, and 13 of Mary Shelley's "Frankenstein." The main character here is the Creature observing the De Lacey family and Safie. The user's query isn't explicit, but they might be looking for an analysis, summary, or help understanding these chapters.
Generated Text: Okay, let's tackle this. The user provided a lengthy excerpt from a novel, specifically Chapters 13 and 14 of Mary Shelley's "Frankenstein." The task is to analyze this text. First, I need to recall the context of these chapters.
Generated Text: Okay, so I need to analyze this text from Mary Shelley's Frankenstein. Let me start by recalling the context. The novel is about Victor Frankenstein creating a monster and the consequences that follow. The passage here is a letter from Elizabeth to Victor, updating him on events back home, followed by Victor's reaction and subsequent events.
Generated Text: Okay, so I need to figure out what the user is asking for here. They provided a lengthy excerpt from what looks like a classic novel, probably Mary Shelley's "Frankenstein" based on the mention of Victor, the creature, and themes like guilt and innocence. The user then included a partial response that ends mid-sentence: "But she has confessed.” This was a dire blow to poor Elizabeth, who had relied with... And then the response stops.
Generated Text: Okay, so I'm trying to understand this passage from Mary Shelley's Frankenstein. Let me start by recalling what I know about the book. The creature created by Victor Frankenstein is narrating his experiences here. He's been rejected by society because of his appearance, and now he's observing a family of cottagers from a distance.
Generated Text: Okay, let's tackle this query. The user provided a lengthy excerpt from Mary Shelley's "Frankenstein," specifically chapters 5 and 6. They included the prompt "ngmhy ess sky" at the beginning, which might be a cipher or code. However, considering the context, it's possible that this is a typo or misinput. The main content is the text from the novel, so the user's request likely relates to analyzing or understanding this passage.
Generated Text: Okay, so I need to analyze this excerpt from Mary Shelley's "Frankenstein." Let me start by recalling the context. This is towards the end of the novel, where Victor Frankenstein is recounting his desperate pursuit of the creature he created. The passage is filled with intense emotion, vivid descriptions of the natural setting, and themes of revenge, isolation, and the consequences of playing God.
Generated Text: Okay, so I need to analyze the provided text. Let me start by reading through it carefully. It looks like excerpts from a novel, possibly "Frankenstein" by Mary Shelley. The letters are from Captain Walton to his sister, Mrs. Saville, and then there's a chapter from the stranger's narrative, who is Victor Frankenstein.
Generated Text: Okay, let's tackle this query. The user provided a lengthy passage from what looks like Mary Shelley's "Frankenstein," specifically the part where Victor Frankenstein is pursuing the creature. The user's query is just "irkxl er." which seems like a typo or random characters. My first thought is that they might have mistyped something related to the text.
Generated Text: Okay, let's tackle this. The user provided a lengthy excerpt from what seems to be Mary Shelley's "Frankenstein," specifically chapters 8 and 9. The task is to analyze this text. First, I need to recall the context of these chapters. Chapter 8 deals with Justine's trial and execution, while Chapter 9 focuses on Victor's guilt and his retreat to nature.
Generated Text: Okay, so I'm trying to understand this passage from Frankenstein. Let me start by summarizing what's happening here. The main character, Victor Frankenstein, is confronted by his creature, who demands that Victor create a companion for him. Victor refuses, leading to a heated argument where the creature threatens to be with Victor on his wedding night. After the creature leaves, Victor decides to dispose of the remains of his previous experiments. He packs his chemical instruments and the remains into a basket, throws them into the sea, and then falls asleep in his boat. When he wakes up, he's drifted far out and ends up landing near a town where he's accused of murdering a man named Henry Clerval. Victor is taken to see the body and collapses in grief, realizing the creature has killed his friend. He then falls into a fever, raving about the deaths he's caused.

additional notes

This is on B200. We've tried a number of variations that don't seem to help:

• Disable cuda graph
• Explicitly set cuda_graph_batch_sizes : [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
• Disable PDL
• Different trtllm versions v0.19.0rc0 and v0.20.0rc1

[laikhtewari]

Thanks for flagging this, Pankaj!

@juney-nvidia this bug in 0.19 RC should be fixed before release

[dsingal0]

We've tried several combinations of configs since and believe the issue is caused by one or more of:

pytorch_backend_config.enable_overlap_scheduler: true
enable_chunked_prefill: true

enabling either deteriorates output quality meaningfully at high bs(32) compared to low bs(2)

Here is the final config that seems to be working well:

pytorch_backend_config:
  enable_overlap_scheduler: false
  cuda_graph_padding_enabled: true
  use_cuda_graph: true
enable_chunked_prefill: false
speculative_config:
  decoding_type: MTP
  num_nextn_predict_layers: 3
kv_cache_config:
  free_gpu_memory_fraction: 0.6
  enable_block_reuse: true

[juney-nvidia]

@dsingal0 @pankajroark

chunked prefill for DS R1 is not supported yet, and for sure here a better error message should be prompted to the users to avoid confusion.

Overlap scheduler should work well and if it can degrade quality then it doesn't match the expectation.

@litaotju @Tracin to help with this issue.

Thanks
June

[ThomasKluiters]

FYI - this also happens with BF16 llama4 - unusable outputs when BS > 1.

Disaling enable_overlap_scheduler worked for me.

[Tracin]

@pankajroark Hi, could you please share the commit ID that can reproduce the issue?

[Tracin]

We do notice an issue about overlap & cuda_graph_padding. It has been fixed in these PRs.
29344e4
dbcfd78
However they are merged into branch release/0.19. Could you try this branch?
And enable_chunked_prefill is not supported for MLA, so please disable it.

[dsingal0]

@Tracin
this still problem still occurs on main as of commit: 1d26a3f Date: Fri May 9 14:51:55 2025 +0800

It also still occurs on release/0.19 as of commit 9727aae Date: Thu May 8 10:25:58 2025 +0800

repro:

pytorch_backend_config:
  enable_overlap_scheduler: true
  cuda_graph_padding_enabled: true
  use_cuda_graph: true
enable_chunked_prefill: false
speculative_config:
  decoding_type: MTP
  num_nextn_predict_layers: 3
kv_cache_config:
  free_gpu_memory_fraction: 0.7
  enable_block_reuse: true

trtllm-serve serve --tp_size 4 --ep_size 4 --max_num_tokens 10000 --max_seq_len 11000 --max_batch_size 128 --backend pytorch --extra_llm_api_options /mnt/data/dsingal/options.yaml /mnt/data/DeepSeek-V3-FP4/             

Then run a high concurrency long context benchmark script against localhost:8000. We see a lot of newlines, simple chinese characters and improper whitespace in the output.
For example:

--------------------------------------------------
Request 119:
Generated Text: broAnalysisletThe

 ** ** and

****TheThe **

 **The.**

1######

 ** ** **

 ,|<<

###

########English:
Hello everyone, welcome to the
"StatQuest" today we're going to talk aboutEnglishEnglishEnglishEnglish###### library
# **THE
COMPLETE
BOOK OF
###### tos
COMP

--------------------------------------------------