Skip to content

Conversation

mattf
Copy link
Collaborator

@mattf mattf commented Sep 6, 2025

What does this PR do?

update Groq inference provider to use OpenAIMixin for openai-compat endpoints

changes on api.groq.com -

Test Plan

$ GROQ_API_KEY=... uv run llama stack build --image-type venv --providers inference=remote::groq --run
...
$ LLAMA_STACK_CONFIG=http://localhost:8321 uv run --group test pytest -v -ra --text-model groq/llama-3.3-70b-versatile tests/integration/inference/test_openai_completion.py -k 'not store'
...
SKIPPED [3] tests/integration/inference/test_openai_completion.py:44: Model groq/llama-3.3-70b-versatile hosted by remote::groq doesn't support OpenAI completions.
SKIPPED [3] tests/integration/inference/test_openai_completion.py:94: Model groq/llama-3.3-70b-versatile hosted by remote::groq doesn't support vllm extra_body parameters.
SKIPPED [4] tests/integration/inference/test_openai_completion.py:73: Model groq/llama-3.3-70b-versatile hosted by remote::groq doesn't support n param.
SKIPPED [1] tests/integration/inference/test_openai_completion.py:100: Model groq/llama-3.3-70b-versatile hosted by remote::groq doesn't support chat completion calls with base64 encoded files.
======================= 8 passed, 11 skipped, 8 deselected, 2 warnings in 5.13s ========================

…-compat functions

changes on api.groq.com -
- json_schema is now supported for specific models, see https://console.groq.com/docs/structured-outputs#supported-models
- response_format with streaming is now supported for models that support response_format
- groq no longer returns a 400 error if tools are provided and tool_choice is not "required"
@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 6, 2025
@raghotham raghotham merged commit d236074 into llamastack:main Sep 6, 2025
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants