[Usage]: Ray + vLLM OpenAI (offline) Batch Inference

[mbuet2ner]

Your current environment

None

How would you like to use vllm

I want to use the OpenAI library to do offline batch inference leveraging Ray (for scaling and scheduling) on top of vLLM.

Context: The plan is to built a FastAPI service that closely mimicks OpenAI's batch API and allows to process a larger number of prompts (tens of thousands) in 24h. There are a few options of achieving this with vLLM but every one has some important drawback, but maybe I am missing something:

• There is an existing guide that uses the LLMClass in the docs with Ray. While the LLMClass shares the OpenAI sampling parameters, it does lack the important OpenAI prompt templating.
• The run_batch.py entrypoint that was introduced here would be the simplest one. But it does not support Ray out of the box.
• The third option would be to use the AsyncLLMEngine as done here and bundle it with OpenAIServingChat as has been done in run_batch.py. But this would entail some (potential) performance degredation due to going asynch even though it is not really needed for offline batch inference.
• The fourth option could be to use Ray serve like in this example from Ray's docs. But this would lack the OpenAI batch format and is – again – async.

Maybe this helps other people as well. Would be super grateful for some feedback. 🙂
And thanks a ton for this very nice piece of software and the great community!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[mbuet2ner]

To answer my own question: The relatively new chat() in the LLMClass does prompt templating/formatting akin to OpenAI. Thus, I can share a working solution (copied from the linked issue):

We are currently leveraging Ray and the LLMClass llm.chat() for that. Really simliar to the very simple generate() example from the docs. v.06.2 #8648 for the llm.chat().

Our current approach is as follows:

• We build a FastAPI application that accepts batch.jsonl files and writes them to a blob storage. You can find the official OpenAI API specification here and can use the OpenAI Pydantic models from the SDK (which are auto-generated from the API spec) to build your endpoints and validate the data.
• The API then leverages Ray via the LLMClass. The llm.chat() has a slightly different interface but the messages format and the sampling parameters are more or less identical to the OpenAI format. Then you can take the Pydantic models from the existing batch entrypoint parse the JSONL files, extract the messages, sampling parameters etc. and give it to llm.chat(). After that you can take the RequestOutput from llm.chat() and iteratively build the BatchRequestOutput and the intermediate OpenAI/vllm-adapted OpenAI Pydantic models.

If there is some easier route please let me know 🙏!