[Core] Add engine option to return only deltas or final output #7381
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Aug 12, 2024
|
This is ready, I'm just planning to add a couple of new tests for it so will hold off adding the ready label for that. |
The LLMEngine and AsyncLLMEngine APIs will currently return/stream cumulative outputs for all sequences at every step. This is more data than needed for LLM.generate or the OpenAI server APIs: - For LLM.generate and non-streaming APIs we only need the final output - For streaming APIs we only require deltas This PR adds an `output_kind` parameter to SamplingParams with an enum value of either CUMULATIVE, DELTA, or FINAL_ONLY. It will reduce the number of objects that need to be constructed at each step, and the amount of data to be serialized to return to the newly-decoupled front-end API process.
joerunde
reviewed
Aug 14, 2024
|
Blocked on some testing issues that I'm trying to solve in #7565 |
6 tasks
# Conflicts: # tests/entrypoints/openai/test_chat.py # vllm/engine/llm_engine.py # vllm/entrypoints/llm.py # vllm/entrypoints/openai/protocol.py # vllm/entrypoints/openai/serving_completion.py # vllm/sampling_params.py
# Conflicts: # vllm/engine/llm_engine.py # vllm/entrypoints/openai/serving_chat.py
# Conflicts: # vllm/engine/llm_engine.py
# Conflicts: # vllm/entrypoints/llm.py
alexm-redhat
approved these changes
Sep 10, 2024
There was a problem hiding this comment.
@njhill thanks for adding the delta request output support, this is really helpful! Left some nit comments, but looks good overall
| raise ValueError( | ||
| "Sampling parameters are missing for a CompletionRequest.") | ||
| finished = seq_group.is_finished() | ||
| if sampling_params.output_kind == RequestOutputKind.FINAL_ONLY and ( |
There was a problem hiding this comment.
Good idea to use sampling_params as the place for output_kind, makes everything simpler.
|
|
||
| outputs = [] | ||
| include_prompt = True | ||
| for seq in top_n_seqs: |
There was a problem hiding this comment.
Good that it also includes the beam search case
Also avoid appending delta token ids to sequences in cases they aren't needed.
# Conflicts: # vllm/sequence.py
# Conflicts: # tests/async_engine/test_async_llm_engine.py
|
Failed tests are unrelated flakes / already failing on main. |
|
Ok cool, LGTM |
simon-mo
approved these changes
Sep 12, 2024
njhill
added a commit
to njhill/vllm
that referenced
this pull request
Sep 13, 2024
Regression in 0.6.1.post1 from vllm-project#7381
This was referenced Sep 13, 2024
Alvant
pushed a commit
to compressa-ai/vllm
that referenced
this pull request
Oct 26, 2024
garg-amit
pushed a commit
to garg-amit/vllm
that referenced
this pull request
Oct 28, 2024
…project#7381) Signed-off-by: Amit Garg <[email protected]>
LeiWang1999
pushed a commit
to LeiWang1999/vllm-bitblas
that referenced
this pull request
Mar 26, 2025
…project#7381) Signed-off-by: LeiWang1999 <[email protected]>
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
LLMEngineandAsyncLLMEngineAPIs will currently return/stream cumulative outputs and prompt-related data for all sequences at every step.This is more data than needed for
LLM.generateor the OpenAI server APIs:This PR adds an
output_kindparameter toSamplingParamswith an enum value of eitherCUMULATIVE,DELTA, orFINAL_ONLY.In the
DELTAcase, data associated with the prompt (prompt token ids, logits, etc.) is returned only in the first output message(s).It will reduce the number of objects that need to be constructed at each step, and the amount of data to be serialized to return to the newly-decoupled front-end API process.