[RFC]: Isolate OpenAI Server Into Separate Process

[robertgshaw2-redhat]

Motivation.

Currently, the OpenAI API server and AsyncLLMEngine share the same asyncio event loop. This means that the API server and the CPU components of the AsyncLLMEngine contend for the same resources. Below, we have a chart of Llama-3-8B running ShareGPT at QPS=10 on an H100.

We can see time is split into three buckets:

• Light blue → this is overlapped GPU execution time (call of execute_model)
• Orange → this is CPU execution time in the LLM engine
• All else → this is API server time

• execute_model does not block the asyncio event loop as it is run in run_in_execuctor

  vllm/vllm/executor/gpu_executor.py

  Line 143 in d92b3c5

  output = await make_async(self.driver_worker.execute_model

• So, we believe that some GPU time is currently already overlapped with the API server

Proposed Change.

Enable better overlapping of the API server and AsyncLLMEngine via multiprocessing by spitting into two processes that communicate over gRPC with protobufs. This is roughly the architecture used by TGI (though TGI has more items in the server than we are proposing here.

Initial Goal

• Write protobuf API to roughly match AsyncLLMEngine generate() method, separate API server into separate process and measure the speedup
  • Will exclude LogitsProcessors - it’s agreed that these don’t belong in SamplingParams anyhow. Instead we’ll include the corresponding external API parameters e.g. for guided decoding
  • Can be a gRPC streaming response, which maps nicely to the stream returned by this method. Cancellations can be mapped to request aborts

Follow Ups

Move items that currently run inside AsyncLLMEngine into the API Server for better overlap with the GPU

• Make necessary changes within the engine to allow full token-id/token-id out so that it can be called such that no tokenization/detokenization is done - you can already pass token ids but there are some places where tokenization/detokenization is done unconditionally such as detokenizing the prompt to return
• Decouple the LogitsProcessor construction from the OpenAI API code - into layer/utility for creating them from the top-level parameters. This can then be called from the gRPC API layer to construct the full SamplingParams
• Decouple incremental detokenization logic so that it can also be used in the API server process
• The API server process can also intercept/process any stop strings, and just abort the request as needed

Feedback Period.

No response

CC List.

@simon-mo @njhill @dsikka @joerunde

Any Other Things.

No response

[mgoin]

Outside of gRPC, ZMQ could be a suitable lightweight alternative that is already used inside vLLM:

vllm/requirements-common.txt

Line 24 in 6a1e25b

pyzmq

Initial PR usage for reference: #6183

[zifeitong]

Outside of gRPC, ZMQ could be a suitable lightweight alternative that is already used inside vLLM:

vllm/requirements-common.txt

Line 24 in 6a1e25b

pyzmq

Initial PR usage for reference: #6183

+1, it's worth doing some benchmark. python_grpc isn't that performant.

Triton Inference Server uses shared memory for commutation.

Edit: Triton Inference Server in fact moved from grpc to shm: triton-inference-server/python_backend#42

[robertgshaw2-redhat]

Sounds good. We are working on a prototype and will measure

shouldn’t be too hard to swap out protocols in the prototype since it’s very simple

[K-Mistele]

Would it be accurate to say that the global interpreter lock is the root cause of the issue here?

[simon-mo]

Would it be accurate to say that the global interpreter lock is the root cause of the issue here?

Pretty much. If we can do freely threaded Python the problem will go away, but the it is still work in progress.

[robertgshaw2-redhat]

Would it be accurate to say that the global interpreter lock is the root cause of the issue here?

Pretty much. If we can do freely threaded Python the problem will go away, but the it is still work in progress.

Its not quite as bad as the chart shows, since GPU execution releases the GIL + some of the front end API server is thus overlapped with GPU execution. But its hard to tell exactly how much thus the experiments we are trying

gRPC is one approach, but there are other ways to add multiprocessing if this does not work

[pzhao1799]

+1, it's worth doing some benchmark. python_grpc isn't that performant.

In fact, in gRPC's own documentation, they claim that streaming RPCs in python are even less performant than unary RPCs. In my experience, this is roughly true so I generally avoid gRPC in python if possible. I think another protocol or leveraging shm are all good things to try, but getting some benchmarked values with gRPC communication (if that's already in the works) will give us somewhere to start.

[robertgshaw2-redhat]

Yeah, it’s a simple prototype to see what the best approach is. Early experiments with grpc aren’t looking great so I’m going to try out zmq tomorrow

[robertgshaw2-redhat]

SUMMARY:

• split openai server into a separate process, such that it does not block the GIL
• leverage zeromq as protocol to implement RRC between openai server process and AsyncLLMEngine process
• leverage protocol buffers as the serialization format

We made three prototypes, separating into separate processes:

• grpc based -> [ DO NOT MERGE ] grpc openai server prototypes #6839
• zeromq based with python pickle -> [ DO NOT MERGE] pyzmq based openai server prototypes (w/ python pickle) #6874
• zeromq based with protobufs -> [ Do Not Merge ] pyzmq based openai server prototypes (w/ protobuf) #6880

zeromq based protocol is delivering speedup in high QPS scenario:

• Llama-3-8B-Instruct
• 1xH100
• 550 input tokens | 150 output tokens
• 10 QPS
• chunked prefill

Metric	main	grpc	zmq + pickle	zmq + protobuf
TPOT (mean)	70.74	64.15	57.35	54.54
TTFT (mean)	687.69	450.05	298.09	269.01

Clear win + clear we should move forward with zeromq

---

Benchmark script for reference:

MODEL="meta-llama/Meta-Llama-3-8B-Instruct"
TOTAL_SECONDS=60
QPS_RATES=("10")

for QPS in ${QPS_RATES[@]}; do
    NUM_PROMPTS=$((TOTAL_SECONDS * QPS))
    echo "===== RUNNING NUM_PROMPTS = $NUM_PROMPTS QPS = $QPS ====="

    python3 benchmarks/benchmark_serving.py \
        --model $MODEL \
        --dataset-name sonnet --sonnet-input-len 550 --sonnet-output-len 150 --dataset-path benchmarks/sonnet.txt \
        --num-prompts $NUM_PROMPTS --request-rate $QPS
done

[vrdn-23]

Apologies if this is not relevant to this issue. Is there a particular reason why vLLM has not considered moving the app server to gunicorn instead of uvicorn?

Uvicorn doesn't do process management and is not recommended to be used for production-grade systems (as mentioned in the uvicorn documentation itself).

I can see that a similar issue was raised at one point (#2573) but it did not seem to get much traction. I was just wondering if this was something that the maintainer team has considered as an option or is this something that can be prioritized hopefully?