[Feature]: NVIDIA Triton GenAI Perf Benchmark

[simon-mo]

🚀 The feature, motivation and pitch

The GenAI perf toolkit from NVIDIA can be used as an alternative benchmark tools for vLLM. While we already have benchmark scripts and framework in benchmarks directory, we should test out different load generators to compare the performance and accuracy of the benchmark clients.

In this issues, I described some tasks that we need help with to try out the new benchmark harness:

• Compare the output of the genai perf with the benchmark_serving, on the coverage of the result metrics and the accuracy.
• Vary the workloads ShareGPT/Sonnet/synthetics
• Implement it as an alternative harness through the script.

Happy to elaborate as well.

https://pypi.org/project/genai-perf/

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[catherinelee274]

would love to work on this. Will investigate the genai perf toolkit

[gitlijian]

Hi, @simon-mo , I really want to participate in solving this problem. I have previously participated in benchmark testing for a competitor in the company. I will update you promptly here if there are any issues or progress.

[catherinelee274]

Are there any good examples of pull requests adding new harnesses?

[simon-mo]

No this will be a new one. @ywang96 added the first harness.

[ganeshku1]

Ideally you should not face any challenges in integrating GenAI-Perf in the vLLM repo for benchmarking.
If you are facing any issues please feel free to tag me here, i will ensure support is provided in a timely manner.
For reference genai-perf documentation with example is here: https://github.com/triton-inference-server/perf_analyzer/blob/main/genai-perf/README.md
There is example of benchmarking vLLM OpenAI endpoint.

[jikunshang]

I submit a PR to add genai-perf benchmark in nightly benchmark. see #10704
feel free to let me know if any requests.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!