[Benchmarking] Add disable_shuffle option for dataset loading

[ymoslem]

Added disable_shuffle argument to control the dataset shuffling behaviour. The option keeps the dataset in the original order in the result to be able to evaluate the responses against the ground truth.

Currently, the dataset is shuffled, which make the requests in a different order than the original dataset. To make evaluation easier, the shuffling should be optional. This disable_shuffle argument disables data shuffling.

The change was tested with a custom dataset where the input is a *.jsonl file.

The main change is modifying:

random.seed(self.random_seed)
random.shuffle(self.data)

to be:

if not getattr(self, 'disable_shuffle', False):
    random.seed(self.random_seed)
    random.shuffle(self.data)

If the change is merged, the new argument should be added to this documentation page.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

[gemini-code-assist]

Code Review

This pull request introduces a disable_shuffle option to control dataset shuffling, which is a valuable addition for ensuring deterministic evaluation. The implementation correctly adds the necessary command-line argument and propagates it to the dataset classes. However, I've identified a critical issue regarding reproducibility. In several dataset loading methods, the random.seed() call has been moved inside the conditional shuffling block. This causes the random number generator to be unseeded when shuffling is disabled, leading to non-reproducible behavior for other random operations within the benchmark. I've provided comments with suggestions to fix this.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

[ywang96]

Looks fine to me!