[Bug]: TypeError: a bytes-like object is required, not 'str'

[xiezhipeng-git]

Your current environment

The output of `python collect_env.py`

Collecting environment information...
PyTorch version: 2.5.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: 14.0.0-1ubuntu1.1
CMake version: version 3.31.2
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov  6 2024, 20:22:13) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.6.56+-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA L4
GPU 1: NVIDIA L4
GPU 2: NVIDIA L4
GPU 3: NVIDIA L4

Nvidia driver version: 560.35.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.6
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               48
On-line CPU(s) list:                  0-47
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) CPU @ 2.20GHz
CPU family:                           6
Model:                                85
Thread(s) per core:                   2
Core(s) per socket:                   24
Socket(s):                            1
Stepping:                             7
BogoMIPS:                             4400.46
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear arch_capabilities
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            768 KiB (24 instances)
L1i cache:                            768 KiB (24 instances)
L2 cache:                             24 MiB (24 instances)
L3 cache:                             38.5 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-47
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown

Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.6.4.1
[pip3] nvidia-cuda-cupti-cu12==12.6.80
[pip3] nvidia-cuda-nvcc-cu12==12.6.85
[pip3] nvidia-cuda-runtime-cu12==12.6.77
[pip3] nvidia-cudnn-cu12==9.6.0.74
[pip3] nvidia-cufft-cu12==11.3.0.4
[pip3] nvidia-curand-cu12==10.3.7.77
[pip3] nvidia-cusolver-cu12==11.7.1.2
[pip3] nvidia-cusparse-cu12==12.5.4.2
[pip3] nvidia-ml-py==12.570.86
[pip3] nvidia-nccl-cu12==2.23.4
[pip3] nvidia-nvcomp-cu12==4.1.0.6
[pip3] nvidia-nvjitlink-cu12==12.6.85
[pip3] onnx==1.17.0
[pip3] optree==0.13.1
[pip3] pynvml==11.4.1
[pip3] pytorch-ignite==0.5.1
[pip3] pytorch-lightning==2.5.0.post0
[pip3] pyzmq==24.0.1
[pip3] sentence-transformers==3.3.1
[pip3] torch==2.5.1+cu121
[pip3] torchaudio==2.5.1+cu121
[pip3] torchinfo==1.8.0
[pip3] torchmetrics==1.6.1
[pip3] torchsummary==1.5.1
[pip3] torchtune==0.5.0
[pip3] torchvision==0.20.1+cu121
[pip3] transformers==4.48.2
[pip3] triton==3.1.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.7.1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	GPU1	GPU2	GPU3	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	PHB	PHB	PHB	0-45	0		N/A
GPU1	PHB	 X 	PHB	PHB	0-45	0		N/A
GPU2	PHB	PHB	 X 	PHB	0-45	0		N/A
GPU3	PHB	PHB	PHB	 X 	0-45	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NVIDIA_VISIBLE_DEVICES=all
CUDA_MINOR_VERSION=2
NVIDIA_REQUIRE_CUDA=cuda>=12.2 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
NCCL_VERSION=2.19.3-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
CUDA_VERSION=12.2.2
LD_LIBRARY_PATH=/usr/local/lib/python3.10/dist-packages/cv2/../../lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
MKL_THREADING_LAYER=GNU
CUDA_MAJOR_VERSION=12
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
CUDA_VISIBLE_DEVICES=0,1,2,3
CUDA_VISIBLE_DEVICES=0,1,2,3
NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
TORCH_NCCL_AVOID_RECORD_STREAMS=1
CUDA_MODULE_LOADING=LAZY
VLLM_USE_V1=1

🐛 Describe the bug

when open v1 has error .no open v1 no error

os.environ["VLLM_USE_V1"] = "1"

MODEL_PATH = "/kaggle/input/qwen2.5/transformers/0.5b-instruct-awq/1"

# MODEL_PATH = snapshot_download('PRIME-RL/Eurus-2-7B-PRIME')
from vllm import LLM, SamplingParams

prompts = ["The future of AI is"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(
    model=MODEL_PATH,
    tensor_parallel_size=1,
    gpu_memory_utilization=0.99,

    # speculative_model=SPECULATIVE_MODEL_PATH,
    # num_speculative_tokens=5,
    use_v2_block_manager=True,
)

outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    print(f"Prompt: {output.prompt!r}, Generated text: {output.outputs[0].text!r}")

TypeError                                 Traceback (most recent call last)
<ipython-input-15-981e7e49d0da> in <cell line: 18>()
     16 sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
     17 
---> 18 llm = LLM(
     19     model=MODEL_PATH,
     20     tensor_parallel_size=1,

/usr/local/lib/python3.10/dist-packages/vllm/utils.py in inner(*args, **kwargs)
   1037                     )
   1038 
-> 1039             return fn(*args, **kwargs)
   1040 
   1041         return inner  # type: ignore

/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/llm.py in __init__(self, model, tokenizer, tokenizer_mode, skip_tokenizer_init, trust_remote_code, allowed_local_media_path, tensor_parallel_size, dtype, quantization, revision, tokenizer_revision, seed, gpu_memory_utilization, swap_space, cpu_offload_gb, enforce_eager, max_seq_len_to_capture, disable_custom_all_reduce, disable_async_output_proc, hf_overrides, mm_processor_kwargs, task, override_pooler_config, compilation_config, **kwargs)
    238         # to avoid import order issues
    239         self.engine_class = self.get_engine_class()
--> 240         self.llm_engine = self.engine_class.from_engine_args(
    241             engine_args, usage_context=UsageContext.LLM_CLASS)
    242 

/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/llm_engine.py in from_engine_args(cls, engine_args, usage_context, stat_loggers, enable_multiprocessing)
     84 
     85         # Create the engine configs.
---> 86         vllm_config = engine_args.create_engine_config(usage_context)
     87         executor_class = Executor.get_class(vllm_config)
     88 

/usr/local/lib/python3.10/dist-packages/vllm/engine/arg_utils.py in create_engine_config(self, usage_context)
   1030                              ) -> VllmConfig:
   1031         if envs.VLLM_USE_V1:
-> 1032             self._override_v1_engine_args(usage_context)
   1033 
   1034         # gguf file needs a specific model loader and doesn't use hf_repo

/usr/local/lib/python3.10/dist-packages/vllm/engine/arg_utils.py in _override_v1_engine_args(self, usage_context)
   1291         from vllm.platforms import current_platform
   1292         device_name = current_platform.get_device_name().lower()
-> 1293         if "h100" in device_name or "h200" in device_name:
   1294             # For H100 and H200, we use larger default values.
   1295             default_max_num_batched_tokens = {

TypeError: a bytes-like object is required, not 'str'

And

from vllm.platforms import current_platform
device_name = current_platform.get_device_name()
print(device_name)

will get

b'NVIDIA L4'

How can I fix it?

@DarkLight1337

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

You should update pynvml to at least version 12.0

[xiezhipeng-git]

You should update pynvml to at least version 12.0

after install nvidia-ml-py-12-570-86 and pynvml 12.0.0
and
Is there still a need to install these again?
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
dask-cuda 24.12.0 requires pynvml<12.0.0a0,>=11.0.0, but you have pynvml 12.0.0 which is incompatible.
dask-cudf-cu12 24.12.0 requires pynvml<12.0.0a0,>=11.4.1, but you have pynvml 12.0.0 which is incompatible.
rapids-dask-dependency 24.12.0 requires pynvml<11.5.0a0,>=11.0.0, but you have pynvml 12.0.0 which is incompatible.
ucx-py-cu12 0.41.0 requires pynvml<12.0.0a0,>=11.4.1, but you have pynvml 12.0.0 which is incompatible.
ucxx-cu12 0.41.0 requires pynvml<12.0.0a0,>=11.4.1, but you have pynvml 12.0.0 which is incompatible.

I can not find ucxx-cu12 and ucxx-cu12 and rapids-dask-dependency on pypi web. And all wheel need <12.0.0a0,>=11.0.0 not 12.0.0
@DarkLight1337

[DarkLight1337]

Can you uninstall both of them, and then only re-install either nvidia-ml-py or pynvml?

[xiezhipeng-git]

Can you uninstall both of them, and then only re-install either nvidia-ml-py or pynvml?

No.I can't.
when I run

%pip install --no-index --find-links=/kaggle/input/pynvml-12-0-0 /kaggle/input/pynvml-12-0-0/pynvml-12.0.0-py3-none-any.whl

I get

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
dask-cuda 24.12.0 requires pynvml<12.0.0a0,>=11.0.0, but you have pynvml 12.0.0 which is incompatible.
dask-cudf-cu12 24.12.0 requires pynvml<12.0.0a0,>=11.4.1, but you have pynvml 12.0.0 which is incompatible.
rapids-dask-dependency 24.12.0 requires pynvml<11.5.0a0,>=11.0.0, but you have pynvml 12.0.0 which is incompatible.
ucx-py-cu12 0.41.0 requires pynvml<12.0.0a0,>=11.4.1, but you have pynvml 12.0.0 which is incompatible.
ucxx-cu12 0.41.0 requires pynvml<12.0.0a0,>=11.4.1, but you have pynvml 12.0.0 which is incompatible.

and install other get

ERROR: Could not find a version that satisfies the requirement pynvml<12.0.0a0,>=11.0.0 (from dask-cuda) (from versions: none)
ERROR: No matching distribution found for pynvml<12.0.0a0,>=11.0.0

And if only install nvidia-ml-py
I also get

TypeError: a bytes-like object is required, not 'str'

[DarkLight1337]

cc @youkaichao

[yuho8818]

same issue. work around for us

[chaunceyjiang]

@xiezhipeng-git What version of nvidia-ml-py do you have?

[xiezhipeng-git]

@xiezhipeng-git What version of nvidia-ml-py do you have?

None and new install nvidia-ml-py==12.570.86

[xiezhipeng-git]

@DarkLight1337 @chaunceyjiang
If the device name is only used to determine whether it is h100 or h200, can I temporarily use the following code first and then wait for the vllm update?

from vllm.platforms import current_platform

# 保存原始的 get_device_name 方法
get_device_name_old = current_platform.get_device_name

def get_device_name_str():
    # 调用原始方法并获取字节类型的设备名称
    device_name_bytes = get_device_name_old()
    # 将字节类型数据解码为字符串
    device_name_str = device_name_bytes.decode('utf-8')
    return device_name_str 

# 如果你需要让 current_platform 的 get_device_name 使用新的字符串返回值版本，
# 可以覆盖原方法，但要注意保持其他属性和方法不变。
# 这里仅替换 get_device_name 方法：
current_platform.get_device_name = get_device_name_str

# 测试新方法
print(current_platform.get_device_name())

[DarkLight1337]

@DarkLight1337 @chaunceyjiang If the device name is only used to determine whether it is h100 or h200, can I temporarily use the following code first and then wait for the vllm update?

Using old version of pynvml in general may cause other errors even if get_device_name initially works. See #12636 (comment)

[xiezhipeng-git]

I didn't try uninstalling pynvml after restarting before . After uninstalling pynvml and installing nvidia-ml-py, there is no such problem these two days. Can observe for a while and then close it.
@DarkLight1337 @chaunceyjiang

[DarkLight1337]

Closing as fixed by #12963