-
Notifications
You must be signed in to change notification settings - Fork 1.2k
fix: Fireworks chat completion broken due to telemetry #3392
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
the fireworks endpoint returns usage information. before making a project wide change, especially one that may result in quietly inconsistent results, a fix must be attempted for the fireworks provider. @raghotham @franciscojavierarceo i recommend reverting and proceeding w/ a fix in the fireworks provider. |
This reverts commit 935b8e2.
@mattf Thanks for the suggestion! Well, it didnt return usage in my test with fireworks provider. (May be I miss something in provider impl, I can have a second look) Would it ok for the api to be broken because the telemetry depends on response to have certain fields? |
api.fireworks.ai returns the info, if it doesn't get propagated then check the fireworks provider. |
# What does this PR do? Fix fireworks chat completion broken due to telemetry expecting response.usage Closes llamastack#3391 ## Test Plan 1. `uv run --with llama-stack llama stack build --distro starter --image-type venv --run` Try ``` curl -X POST http://0.0.0.0:8321/v1/openai/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "fireworks/accounts/fireworks/models/llama-v3p1-8b-instruct", "messages": [{"role": "user", "content": "Hello!"}] }' ``` ``` {"id":"chatcmpl-ee922a08-0df0-4974-b0d3-b322113e8bc0","choices":[{"message":{"role":"assistant","content":"Hello! How can I assist you today?","name":null,"tool_calls":null},"finish_reason":"stop","index":0,"logprobs":null}],"object":"chat.completion","created":1757456375,"model":"fireworks/accounts/fireworks/models/llama-v3p1-8b-instruct"}% ``` Without fix fails as mentioned in llamastack#3391 Co-authored-by: Francisco Arceo <[email protected]>
What does this PR do?
Fix fireworks chat completion broken due to telemetry expecting response.usage
Closes #3391
Test Plan
uv run --with llama-stack llama stack build --distro starter --image-type venv --run
Try
Without fix fails as mentioned in #3391