Input should be a valid dictionary or instance of StatusBreakdown

[pliops-daniels]

Hi,
I installed guidellm from source (commit hash 9108369).

I used vLLM to serve a model (llama 3.1 8B) and ran this guidellm command:

export MODEL_NAME=/hf_models/meta-llama/Meta-Llama-3.1-8B-Instruct/

guidellm benchmark \
  --target "http://localhost:8008" \
  --model $MODEL_NAME \
  --rate-type concurrent \
  --rate 4.0 \
  --max-seconds 15 \
  --data "prompt_tokens=256,output_tokens=16"

The benchmark seems to run for 15 seconds, but then fails:

pydantic_core._pydantic_core.ValidationError: 1 validation error for BenchmarkRunStats
requests_made
  Input should be a valid dictionary or instance of StatusBreakdown[int, int, int, int] [type=model_type, 
input_value=StatusBreakdown(successfu...incomplete=4, total=144), input_type=StatusBreakdown]
    For further information visit https://errors.pydantic.dev/2.9/v/model_type

The python/pydantic version I use:

python --version
Python 3.10.12

pip show pydantic
Name: pydantic
Version: 2.9.2

[jaredoconnell]

I tested this with the following Python and pydantic versions:

[root@ubi-playground guidellm]# python3 --version
Python 3.9.21
[root@ubi-playground guidellm]# pip show pydantic
Name: pydantic
Version: 2.11.7

And I did not get that validation error on both main and on the same commit, with the only major difference being that I used a different model. Your pydantic install is 2 minor versions behind. I wonder if that has anything to do with it. Can you retest this with a more modern version of pydantic?
Additional context: Mine also runs for about 15 seconds. The error appears to be related to the output processing.

[pliops-daniels]

I created a new python venv (python 3.10.12) with pydantic-2.11.7 and install guidellm (commit hash 9108369).
The guidellm command worked.

I then downgraded pydantic:

pip install pydantic==2.9.2
...
      Successfully uninstalled pydantic-2.11.7
Successfully installed pydantic-2.9.2 pydantic-core-2.23.4

And got the error again:

    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for BenchmarkRunStats

I then upgraded pydantic back to a new version:

pip install pydantic --upgrade
...
      Successfully uninstalled pydantic-2.9.2
Successfully installed pydantic-2.11.7 pydantic-core-2.33.2

And the command worked again.

So your observation seems to be correct, the issue seems to be related to pydantic-2.9.2.