[Feature]: Add support for multi-lora and single lora for classification tasks

[power-puff-gg]

🚀 The feature, motivation and pitch

Are there any plans to introduce support for multi-lora and single lora for classification tasks?

I am trying to use Phi3.5 instruct model with classification head as a custom model. I also tried to add support for lora using the SupportsLoRA interface with the lora supported modules declarations but it seems to be of no use. Curious if this feature is still being considered by the vLLM team or not. If yes, what is the tentative timeline for it?

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[jeejeelee]

I will make a related investigation and then reply.

[PancakeAwesome]

Could you please tell me when this function will be updated? Currently, our multimodal classification model combined with Lora cannot use VLLM for inference.

[jeejeelee]

Sorry, I have no enough bandwidth to inverisigate this issue, I'll first add a dd label - maybe someone will be willing to implement it. I'll consider this issue later

[sfeng33]

Hey @power-puff-gg , to clarify, are you hoping to make lora work with Phi3.5 instruct? You don't need multimodal, correct? I might be able to help.

[power-puff-gg]

Hi @sfeng33 ,Yes basically we wanted to use multi-lora setup for classification tasks. I believe it is not supported by vLLM as of today, am I wrong?

[sfeng33]

Yes the phi model is not supported at the moment, I can help looking into the implementation.

[power-puff-gg]

Thank you! Appreciate it very much.

Also I noticed while creating a plugin for Phi3.5, the vLLM implementation of Phi3 model is a bit confusing as it relies on Llama and does not have much code to refer to. Pardon me if I am missing some understanding here.

[sfeng33]

@power-puff-gg do you have a lora I can test with?

[power-puff-gg]

I do not have it handy. I am sorry.

[sfeng33]

Hey @power-puff-gg, here is my understanding on what you want to achieve:

1. Add a classification head (score layer) to the base model
   This can be done via

  python -m vllm.entrypoints.openai.api_server \
    --model microsoft/Phi-3-mini-4k-instruct \
    --convert classify \
    --runner pooling \
    --load-format dummy \
    --override-pooler-config '{"pooling_type": "LAST", "normalize": false, "softmax": true}' 

2. Load a LoRA that also contains the trained head
   e.g.

    --enable-lora 
    --lora-modules ....

The phi3.5 model is actually lora supported. The challenge is rather on #2, where vllm doesn't support layer replacement (the classification/score layer), context: #15509.
Though alternatively I think you can have a merged model so that there is no lora needed.

[power-puff-gg]

Hi @sfeng33 , thanks for looking further into this. You are right, vllm doesn't support layer replacement (the classification/score layer). But I have a custom classification layer as a LAST layer for my use case.

I do not want to merge the lora adapter to the model as I want to load and unload multiple lora adapters on the fly for various use cases with the same base model. Hence merging is not an option for me.

Also I noticed while working on the model, Phi3.5 was not working with --enable-lora and --classify flags set together.

This is the original model I was trying to use lora for: https://github.com/power-puff-gg/custom_model_vllm/blob/main/original_simple_model.py. You can also find the code I created for model plugin registration as the classification layers were different.

[sfeng33]

So is the issue that your custom model doesn't work with lora?