Respect passed in device overrides in engine args #18808
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
|
This pull request was exported from Phabricator. Differential Revision: D75496412 |
Summary: Signed-off-by: Adolfo Victoria [email protected] When calling `create_engine_config` we don't respect the device set in engine args and instead grab it from the runtime platform directly. This logic is also extraneous since the default behavior (i.e. device=auto) is to set the device to the runtime platform device if auto is passed inside of `__post_init__` inside of `DeviceConfig` Differential Revision: D75496412
Adolfo-Karim
force-pushed
the
export-D75496412
branch
from
32d6746 to
424b3dc
Compare
|
This pull request was exported from Phabricator. Differential Revision: D75496412 |
| current_platform.pre_register_and_update() | ||
|
|
||
| device_config = DeviceConfig(device=current_platform.device_type) | ||
| device_config = DeviceConfig(device=self.device) |
There was a problem hiding this comment.
We have already deprecated the --device argument in #18399 after multiple platform plugin authors agreed to it. Could you elaborate on how this is used in your application? cc @youkaichao
There was a problem hiding this comment.
Seems we intentionally make "device" arg not honored. Wondering @Adolfo-Karim , do we need to use "device" in EngineArg for something?
There was a problem hiding this comment.
Discuss offline with @Adolfo-Karim , it's more like a "CPU" deterministic logic.
There was a problem hiding this comment.
"CPU" deterministic logic
what does this mean?
There was a problem hiding this comment.
https://github.com/vllm-project/vllm/blob/main/vllm/platforms/__init__.py#L156, here, we check the "CPU" in the package name. We don't actually have package, since we use monolithic / unified codebase.
|
I guess we probably can close this PR per discussion with @Adolfo-Karim , likely we will solve this issue with internal workaround. |
Signed-off-by: Adolfo Victoria [email protected]
Summary: When calling
create_engine_configwe don't respect the device set in engine args and instead grab it from the runtime platform directly. This logic is also extraneous since the default behavior (i.e. device=auto) is to set the device to the runtime platform device if auto is passed inside of__post_init__inside ofDeviceConfigDifferential Revision: D75496412