feat: add batches API with OpenAI compatibility #3088
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add complete batches API implementation with protocol, providers, and tests: Core Infrastructure: - Add batches API protocol using OpenAI Batch types directly - Add Api.batches enum value and protocol mapping in resolver - Add OpenAI "batch" file purpose support - Include proper error handling (ConflictError, ResourceNotFoundError) Reference Provider: - Add ReferenceBatchesImpl with full CRUD operations (create, retrieve, cancel, list) - Implement background batch processing with configurable concurrency - Add SQLite KVStore backend for persistence - Support /v1/chat/completions endpoint with request validation Comprehensive Test Suite: - Add unit tests for provider implementation with validation - Add integration tests for end-to-end batch processing workflows - Add error handling tests for validation, malformed inputs, and edge cases Configuration: - Add max_concurrent_batches and max_concurrent_requests_per_batch options - Add provider documentation with sample configurations Test with - ``` $ uv run llama stack build --image-type venv --providers inference=YOU_PICK,files=inline::localfs,batches=inline::reference --run & $ LLAMA_STACK_CONFIG=http://localhost:8321 uv run pytest tests/unit/providers/batches tests/integration/batches --text-model YOU_PICK ```
mattf
requested review from
ashwinb,
yanxi0830,
hardikjshah,
raghotham,
ehhuang,
terrytangyuan,
leseb,
bbrowning,
reluctantfuturist and
slekkala1
as code owners
meta-cla
bot
added
the
CLA Signed
|
lgtm I am not a fan of the "batches" name for an API but hey this time I can just ignore that piece and point to someone else :) |
ashwinb
approved these changes
Aug 12, 2025
this also captures other notes from agents, eval and inference apis
i updated provider_codegen.py to include the api protocol's docstring in the providers docs and added a note about the development status. fyi, this also picked up additional docs from agents, inference & eval. |
|
@ashwinb thoughts on the replay tests failing? |
| batch_id = f"batch_{uuid.uuid4().hex[:16]}" | ||
| current_time = int(time.time()) | ||
|
|
||
| batch = BatchObject( |
There was a problem hiding this comment.
It could be useful to generate an idempotency key for retries and avoiding processing the same file more than once.
There was a problem hiding this comment.
❤️ idempotency is hugely valuable for apis like this.
since files are immutable, the params can serve as that id.
let me add an issue for that feature.
There was a problem hiding this comment.
since files are immutable, the params can serve as that id.
💯
ashwinb
added a commit
that referenced
this pull request
Aug 14, 2025
Reverts #3088 The PR broke integration tests.
cdoern
pushed a commit
to cdoern/llama-stack
that referenced
this pull request
Aug 14, 2025
Add complete batches API implementation with protocol, providers, and tests: Core Infrastructure: - Add batches API protocol using OpenAI Batch types directly - Add Api.batches enum value and protocol mapping in resolver - Add OpenAI "batch" file purpose support - Include proper error handling (ConflictError, ResourceNotFoundError) Reference Provider: - Add ReferenceBatchesImpl with full CRUD operations (create, retrieve, cancel, list) - Implement background batch processing with configurable concurrency - Add SQLite KVStore backend for persistence - Support /v1/chat/completions endpoint with request validation Comprehensive Test Suite: - Add unit tests for provider implementation with validation - Add integration tests for end-to-end batch processing workflows - Add error handling tests for validation, malformed inputs, and edge cases Configuration: - Add max_concurrent_batches and max_concurrent_requests_per_batch options - Add provider documentation with sample configurations Test with - ``` $ uv run llama stack build --image-type venv --providers inference=YOU_PICK,files=inline::localfs,batches=inline::reference --run & $ LLAMA_STACK_CONFIG=http://localhost:8321 uv run pytest tests/unit/providers/batches tests/integration/batches --text-model YOU_PICK ``` addresses llamastack#3066
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add complete batches API implementation with protocol, providers, and tests:
Core Infrastructure:
Reference Provider:
Comprehensive Test Suite:
Configuration:
Test with -
addresses #3066