[Feature]: Only apply Guided/Structured grammar after reasoning steps in Reasoning models

[cksac]

🚀 The feature, motivation and pitch

Only apply Guided/Structured grammar only in the answer for reasoning model. i.e. for DeepSeek R1 only enforce grammar inside <answer></answer> or after </think>
This would make Reasoning models more useful in agent workflow expecting structured output.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[gaocegege]

Related to #11908

[gaocegege]

I have a PoC tested with deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B, which introduces a checker in the logits processor to skip xgrammar schema check:

class XGrammarLogitsProcessor:
    """Wrapper class to support pickle protocol"""
    config: GrammarConfig

    ctx: xgr.CompiledGrammar | None = None
    token_bitmask: torch.Tensor = None  # type: ignore[assignment]
    matchers: list[xgr.GrammarMatcher] = field(default_factory=list)
    batch_size: int = field(default=1)
    prefilled: bool = field(default=False)

    def __call__(self, input_ids: list[int],
                 scores: torch.Tensor) -> torch.Tensor:
+        if not reasoning_end(input_ids):
+           return scores
        if self.ctx is None:
            self._ensure_ctx()

def reasoning_end(input_ids: list[int]) -> bool:
    """Check if the input_ids contain the end of reasoning token."""
    # Hard coded endthink token id </think>
    endthink_token_id = 151649

    if endthink_token_id in input_ids:
        return True

I can generalize it to support all R1 models and structured engines if you think this approach is effective.

full code:

gaocegege@9134347

[gaocegege]

Having a draft PR, #12955

Please let me know if the approach works for you.

[gaocegege]

Hi @cksac

This should be ready in latest main branch. You could give it a try. https://docs.vllm.ai/en/latest/getting_started/examples/openai_chat_completion_structured_outputs_with_reasoning.html

[gaocegege]

#14113 There is a known issue. FYI

Currently only xgrammar is supported.