[Bug]: v0.4.1 VLLM_USE_MODELSCOPE not working

[kratorado]

Your current environment

The output of `python collect_env.py`

🐛 Describe the bug

the related code in the previous version :

if os.environ.get("VLLM_USE_MODELSCOPE", "False").lower() == "true":

now the code:
if VLLM_USE_MODELSCOPE:
but the VLLM_USE_MODELSCOPE is set to string "true" by

VLLM_USE_MODELSCOPE = os.environ.get("VLLM_USE_MODELSCOPE",
                                     "False").lower() == "true"

so, VLLM_USE_MODELSCOPE will never work

[simon-mo]

sorry why would it never work? if the env var is set VLLM_USE_MODELSCOPE=true then the statement would evaluate to true?

[kratorado]

sorry why would it never work? if the env var is set VLLM_USE_MODELSCOPE=true then the statement would evaluate to true?

ah, I reread the code and found that I made a wrong debugging result.
Actually what I met is that when I run docker image vllm/vllm-openai:v0.4.1 , passing env with --env "VLLM_USE_MODELSCOPE=True" , vllm still visits huggingface.co to download models. It seems that vllm doesnot take this env var into account.

[kratorado]

My temp workaround is adding the origin snippet to vllm/config.py line 106:

...
        if VLLM_USE_MODELSCOPE:
            # download model from ModelScope hub,
            # lazy import so that modelscope is not required for normal use.
            # pylint: disable=C.
            from modelscope.hub.snapshot_download import snapshot_download

            if not os.path.exists(model):
                model_path = snapshot_download(model_id=model, revision=revision)
            else:
                model_path = model
...

[simon-mo]

can you send a PR for what worked for you?

[kratorado]

Sending a PR is easy. But my workaround is reverting some changes of a refactor which I think is the cause, I have to read the whole refactored code to confirm what to do is best.
related PR #4097

[kratorado]

vllm/config.py , line 107

self.hf_config = get_config(self.model, trust_remote_code, revision,
                                    code_revision)

calls vllm/transformers_utils/config.py , line 23,

        config = AutoConfig.from_pretrained(
            model,
            trust_remote_code=trust_remote_code,
            revision=revision,
            code_revision=code_revision)

The code above does not check the environment variable VLLM_USE_MODELSCOPE and it will download config file from huggingface.co by default.

Have no clue how to fix it elegantly now.

[dashi6174]

My friends，how to fix it ?

[kratorado]

My friends，how to fix it ?

insert the code, at the line 106 of the file vllm/config. before self.hf_config = get_config(self.model, trust_remote_code, revision, code_revision)

        if VLLM_USE_MODELSCOPE:
            from modelscope.hub.snapshot_download import snapshot_download

            if not os.path.exists(model):
                model_path = snapshot_download(model_id=model,
                                               revision=revision)
            else:
                model_path = model
            self.model = model_path
            self.download_dir = model_path
            self.tokenizer = model_path

Remember it is not the best way, just a temp workaround.