Multi-node serving with vLLM - Problems with Ray

[vbucaj]

I am trying to run a distributed (multi-node) inference server with vLLM using ray, but I keep getting the following ValueError:

Ray does not allocate any GPUs on the driver node. Consider adjusting the Ray placement group or running the driver on a GPU node.

I'm not sure how exactly to resolve this. I suspect the issue is with this script https://github.com/vllm-project/vllm/blob/main/vllm/engine/ray_utils.py, especially when a ray_address is passed. Is there a specific ray_address arg that gets passed during the ray.init() stage?

More specifically, it seems like this error is raised because of the driver_dummy_worker in line 182 of https://github.com/vllm-project/vllm/blob/main/vllm/engine/llm_engine.py.

I'm confused what's going on with this piece of code

   def _init_workers_ray(self, placement_group: "PlacementGroup",
                          **ray_remote_kwargs):
        if self.parallel_config.tensor_parallel_size == 1:
            num_gpus = self.cache_config.gpu_memory_utilization
        else:
            num_gpus = 1

        self.driver_dummy_worker: RayWorkerVllm = None
        self.workers: List[RayWorkerVllm] = []

        driver_ip = get_ip()
        for bundle_id, bundle in enumerate(placement_group.bundle_specs):
            if not bundle.get("GPU", 0):
                continue
            scheduling_strategy = PlacementGroupSchedulingStrategy(
                placement_group=placement_group,
                placement_group_capture_child_tasks=True,
                placement_group_bundle_index=bundle_id,
            )
            worker = ray.remote(
                num_cpus=0,
                num_gpus=num_gpus,
                scheduling_strategy=scheduling_strategy,
                **ray_remote_kwargs,
            )(RayWorkerVllm).remote(self.model_config.trust_remote_code)

            worker_ip = ray.get(worker.get_node_ip.remote())
            if worker_ip == driver_ip and self.driver_dummy_worker is None:
                # If the worker is on the same node as the driver, we use it
                # as the resource holder for the driver process.
                self.driver_dummy_worker = worker
            else:
                self.workers.append(worker)

        if self.driver_dummy_worker is None:
            raise ValueError(
                "Ray does not allocate any GPUs on the driver node. Consider "
                "adjusting the Ray placement group or running the driver on a "
                "GPU node.")

When the error is raised, it is checking if driver_dummy_worker is None, but don't we set it to None above, that is, self.driver_dummy_worker: RayWorkerVllm =None?

[nkwangleiGIT]

you can have a try following doc here: http://kubeagi.k8s.com.cn/docs/category/distributed-inference

[jens5588]

have you solved the problem? have the same error with multi-node inference

[nkwangleiGIT]

@jens5588 did you solve it? here is what maybe helpful for you.

Check the default spec of placement group:

vllm/vllm/executor/ray_utils.py

Line 110 in eefeb16

placement_group_specs = ([{"GPU": 1}] * parallel_config.world_size)

AFAIK, it'll work normally if the driver process and bundle are on the same node, so the serving may restart if it's not the case, and will retry if it's not the case.
You can also print out the driver and worker ip to check whether it's the case, you can also check the placement group where the bundle is.

And there is another issue recently if you're on recent build: https://github.com/vllm-project/vllm/pull/2727/files

[atanikan]

I see the same issue when tp>1 . Do we have a fix for this?

[liurupeng]

Saw the same issue, any workaround? and what is the reason?

[youkaichao]

If you don't use more GPUs than one node in one vLLM instance, then you can use the mp backend: --distributed-executor-backend mp . see https://docs.vllm.ai/en/stable/serving/distributed_serving.html for more details.

[liurupeng]

@youkaichao my case is that I used two GPU nodes (each one has 8 GPUs), and my two containers (ray head and ray worker) both used all the 8 GPU. Is there a workaround for this as well?

[youkaichao]

then why do you need to organize them into a ray cluster? they can work individually.

[liurupeng]

is there a way to deploy one vlllm instance in two ray clusters?

[gabrielmbmb]

I had the same issue. In my case, I have a two nodes Ray cluster with 8XH100 each. I was creating a Ray actor that was the driver creating the vllm.LLM with tensor_parallel_size==8. What it was happening is that the driver IP was different than the worker IP where the RayWorkerWrapper actors were being allocated. What I did to solve the issue is to create a PlacementGroup with the GPU bundles required and with the STRICT_PACK strategy and the _soft_target_node_id set to one node, and use this placement group with my Ray actor. As vLLM takes the current placement group and assign it to the RayWorkerWrapper actor that it creates, this assures that the driver and the workers are on the same node, thus avoiding the exception.

https://github.com/argilla-io/distilabel/blob/9aa8b35308dc82fd1f9152743e1057db449b0169/src/distilabel/pipeline/ray.py#L267-L316