"llm.generate()" API support Continuous Batching?

[weishengying]

In my understanding, Continuous Batching is a trade-off between request latency and throughput, it can return the results of the completed request in time instead of waiting all requests of a batch. But it is seemingly that the “llm.generate()” API alaways wait all requests of a batch have been completed before return.

[nlpkiddo-2001]

Hi , I am new to this vLLM, I need to make batch calls in vllm
prompts = ["Once upon a time .."] * 10

Is vllm has native support for this? and if so is this good approach as like sending individual request concurrently?
what would be tradeoff here?

thanks in advance