Support for /v1/batches endpoint following the OpenAI API specification #3066
Description
🚀 Describe the new functionality needed
This feature request proposes the implementation of the /v1/batches endpoint as defined in the OpenAI API specification. The /v1/batches endpoint enables users to submit large volumes of requests asynchronously for processing, rather than making individual synchronous API calls.
The key functionalities to implement include:
- Batch Submission (POST /v1/batches) – Accepts a JSONL file containing a list of individual API requests, along with metadata describing the type of request (e.g. completions, chat, etc.), a completion webhook, and processing configuration.
- Batch Status Retrieval (GET /v1/batches/{batch_id}) – Provides real-time updates on the batch status (validating, in_progress, completed, failed, etc.).
- Batch Listing (GET /v1/batches) – Allows users to list and filter their historical batch jobs.
💡 Why is this needed? What if we don't build it?
Implementing batch support is critical for scaling usage efficiently and improving developer experience. Specifically:
- Efficiency & Performance: Users handling high volumes of similar requests (e.g. evaluations, fine-tuning tests, bulk generations) currently rely on looping through synchronous calls, leading to rate-limiting issues, reduced performance, and brittle retry logic.
- Reliability: The batch endpoint allows for robust job handling, including automatic retries, tracking, and clean separation of request lifecycle management.
- Cost & Resource Optimization: Batching enables better backend optimization and resource planning compared to handling thousands of individual requests.
- Developer Experience: Aligning with OpenAI’s own API spec improves cross-compatibility, reduces friction for users migrating or integrating across platforms, and helps maintain API parity.
Other thoughts
No response