Support denvr endpoints with Litellm. #2085
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 3 commits
June 20, 2025 02:52
Signed-off-by: Ubuntu <azureuser@denvr-inf.kifxisxbiwme5gt4kkwqsfdjuh.dx.internal.cloudapp.net>
Signed-off-by: Ubuntu <azureuser@denvr-inf.kifxisxbiwme5gt4kkwqsfdjuh.dx.internal.cloudapp.net>
Signed-off-by: Ubuntu <azureuser@denvr-inf.kifxisxbiwme5gt4kkwqsfdjuh.dx.internal.cloudapp.net>
srinarayan-srikanthan
requested review from
lvliang-intel and
letonghan
as code owners
Dependency Review✅ No vulnerabilities or license issues found.Scanned FilesNone |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
for more information, see https://pre-commit.ci
Signed-off-by: Ubuntu <azureuser@denvr-inf.kifxisxbiwme5gt4kkwqsfdjuh.dx.internal.cloudapp.net>
https://github.com/srinarayan-srikanthan/GenAIExamples into denvr_chat merging remote
There was a problem hiding this comment.
Pull Request Overview
Adds support for deploying ChatQnA with remote denvr inference endpoints and updates the streaming response parser for multi-chunk JSON outputs.
- Introduce a new
compose_remote.yamlworkflow and environment variable instructions in the Xeon CPU Docker README. - Update
align_generatorinchatqna.pyto split and process multiple JSON chunks per line.
Reviewed Changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| ChatQnA/docker_compose/intel/cpu/xeon/README.md | Added remote endpoint deployment steps and updated compose table |
| ChatQnA/chatqna.py | Refactored align_generator to handle multi-chunk streaming JSON |
Comments suppressed due to low confidence (3)
ChatQnA/docker_compose/intel/cpu/xeon/README.md:78
- [nitpick] Clarify whether
REMOTE_ENDPOINTshould include the/v1/chat/completionspath or just the base URL to avoid confusion.
**Note**: Set REMOTE_ENDPOINT variable value to "https://api.inference.denvrdata.com" when the remote endpoint to access is "https://api.inference.denvrdata.com/v1/chat/completions"
ChatQnA/chatqna.py:178
- [nitpick] The outer variable
lineis reused for the inner loop below, which can reduce readability; consider renaming the loop variable tochunkor similar.
chunks = [chunk.strip() for chunk in line.split("\n\n") if chunk.strip()]
ChatQnA/chatqna.py:191
- The previous
finish_reasoncheck was removed, which may cause tokens to be emitted after the stream should end; consider re-adding or documenting this behavior change.
elif "content" in json_data["choices"][0]["delta"]:
Signed-off-by: Ubuntu <azureuser@denvr-inf.kifxisxbiwme5gt4kkwqsfdjuh.dx.internal.cloudapp.net>
Signed-off-by: Ubuntu <azureuser@denvr-inf.kifxisxbiwme5gt4kkwqsfdjuh.dx.internal.cloudapp.net>
for more information, see https://pre-commit.ci
Signed-off-by: Ubuntu <azureuser@denvr-inf.kifxisxbiwme5gt4kkwqsfdjuh.dx.internal.cloudapp.net>
…/GenAIExamples into denvr_chat merging local to remote
for more information, see https://pre-commit.ci
Signed-off-by: Ubuntu <azureuser@denvr-inf.kifxisxbiwme5gt4kkwqsfdjuh.dx.internal.cloudapp.net>
…/GenAIExamples into denvr_chat merging divergent remote
for more information, see https://pre-commit.ci
Signed-off-by: Ubuntu <azureuser@denvr-inf.kifxisxbiwme5gt4kkwqsfdjuh.dx.internal.cloudapp.net>
srinarayan-srikanthan
requested review from
jaswanth8888 and
hteeyeoh
as code owners
for more information, see https://pre-commit.ci
35 tasks
alexsin368
reviewed
Jun 27, 2025
| no_proxy: ${no_proxy} | ||
| http_proxy: ${http_proxy} | ||
| https_proxy: ${https_proxy} | ||
| LLM_ENDPOINT: ${LLM_ENDPOINT} |
There was a problem hiding this comment.
should this be:
LLM_ENDPOINT: ${REMOTE_ENDPOINT}
louie-tsai
reviewed
Jun 27, 2025
|
|
||
| ```bash | ||
| export REMOTE_ENDPOINT=<endpoint-url> | ||
| export LLM_MODEL_ID=<model-id> |
There was a problem hiding this comment.
good to put a notice why we need to set LLM_MODEL_ID again and users need to pick one supported by remote endpoint.
| ```bash | ||
| export REMOTE_ENDPOINT=<endpoint-url> | ||
| export LLM_MODEL_ID=<model-id> | ||
| export OPENAI_API_KEY=<API-KEY> |
There was a problem hiding this comment.
good to change it from OPENAI_API_KEY to API_KEY since it is not for openai
| - **To Run:** | ||
|
|
||
| ```bash | ||
| export OPENAI_API_KEY=<api-key>> |
There was a problem hiding this comment.
OPENAI name here is confusing. good to remove OPENAI term here.
|
|
||
| ```bash | ||
| export LLM_ENDPOINT=<endpoint-url> | ||
| export LLM_MODEL_ID=<model-id> |
There was a problem hiding this comment.
Better to mention how users to set LLM_MODEL_ID correctly by getting the supported model list
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Support remote inference with denvr endpoint for chatqna with readme updates.
Issues
#2084
Type of change
List the type of change like below. Please delete options that are not relevant.
Dependencies
N/A
Tests
Describe the tests that you ran to verify your changes.