[Usage]: add mulitple lora in docker

[chintanshrinath]

Your current environment

The output of `python collect_env.py`

How would you like to use vllm

Hi
I want to attach lora using docker command

docker run --runtime nvidia --gpus all
-v ~/.cache/huggingface:/root/.cache/huggingface
-v /datadrive/finetune_model/infosys:/app/lora/xyz
-v /datadrive/finetune_model/dummy:/app/lora/abc
-p 8000:8000
--env "HUGGING_FACE_HUB_TOKEN="
vllm/vllm-openai --enable-lora
--model meta-llama/Meta-Llama-3-8B-Instruct
--lora-modules xyz-lora=/datadrive/finetune_model/xyz
--lora-modules abc-lora=/datadrive/finetune_model/abc

However, I am getting below error

{'object': 'error', 'message': 'The model `xyz-lora` does not exist.', 'type': 'NotFoundError', 'param': None, 'code': 404}

Can anyone help here? Did I use wrong command?

Thanks you

[youkaichao]

I think the lora path should be the path inside the container, e.g. --lora-modules xyz-lora=/app/lora/xyz

[chintanshrinath]

Hi @youkaichao
Thank you for reply

Something like that
--lora-modules xyz-lora=vllm/vllm-openai/app/lora/xyz?

[youkaichao]

vllm/vllm-openai is the image, not the path.

/app/lora/xyz should work.

you can ask chatgpt for more details.

[Cloopen-ReLiNK]

0.5.4 docker api_server.py: error: unrecognized arguments: --lora-modules test-lora=/vllm-workspace/xxxx/

[chintanshrinath]

Hi @Cloopen-ReLiNK
docker run --runtime nvidia --gpus all
-v ~/.cache/huggingface:/root/.cache/huggingface
-v /datadrive/finetune_model/inf:/app/lora/inf
-v /datadrive/finetune_model/dummy:/app/lora/dummy
-p 8000:8000
--env "HUGGING_FACE_HUB_TOKEN=sxy"
vllm/vllm-openai --enable-lora
--model meta-llama/Meta-Llama-3-8B-Instruct
--lora-modules inf-lora=/app/lora/inf dummy-lora=/app/lora/dummy

I am able to run with above command. Please try with this one

[SMAntony]

Is it just me or does vLLM docker documentation is not extensive? Like I couldnt find what other command line arguments you can use.

[hmellor]

Closing as this issue appears to be resolved.