[Feature]: Support for Seq classification/Reward models

[ariaattar]

🚀 The feature, motivation and pitch

Verifier/reward models are going to be very important moving forward for building:

• High quality synthetic data pipelines
• Verifying model reasoning
• Multi agent systems

Could we add support for sequence classification models like Skywork/Skywork-Reward-Llama-3.1-8B

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[youkaichao]

contribution is welcome!

vllm already supports embedding model. I think it is quite similar to reward model. I don't know what would be the obstacle to use vllm code to run reward models. We can pretend they are embedding models.

[ariaattar]

Seems like a lot of the reward models use a different architecture than embedding models.

ValueError: Model architectures ['Gemma2ForSequenceClassification'] are not supported for now.
ValueError: Model architectures ['LlamaForSequenceClassification'] are not supported for now. 

 Supported architectures: ['AquilaModel', 'AquilaForCausalLM', 'BaiChuanForCausalLM', 'BaichuanForCausalLM', 'BloomForCausalLM', 'ChatGLMModel', 'ChatGLMForConditionalGeneration', 'CohereForCausalLM', 'DbrxForCausalLM', 'DeciLMForCausalLM', 'DeepseekForCausalLM', 'DeepseekV2ForCausalLM', 'ExaoneForCausalLM', 'FalconForCausalLM', 'GemmaForCausalLM', 'Gemma2ForCausalLM', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GPTJForCausalLM', 'GPTNeoXForCausalLM', 'InternLMForCausalLM', 'InternLM2ForCausalLM', 'JAISLMHeadModel', 'LlamaForCausalLM', 'LLaMAForCausalLM', 'MistralForCausalLM', 'MixtralForCausalLM', 'QuantMixtralForCausalLM', 'MptForCausalLM', 'MPTForCausalLM', 'MiniCPMForCausalLM', 'NemotronForCausalLM', 'OlmoForCausalLM', 'OPTForCausalLM', 'OrionForCausalLM', 'PersimmonForCausalLM', 'PhiForCausalLM', 'Phi3ForCausalLM', 'PhiMoEForCausalLM', 'Qwen2ForCausalLM', 'Qwen2MoeForCausalLM', 'Qwen2VLForConditionalGeneration', 'RWForCausalLM', 'StableLMEpochForCausalLM', 'StableLmForCausalLM', 'Starcoder2ForCausalLM', 'ArcticForCausalLM', 'XverseForCausalLM', 'Phi3SmallForCausalLM', 'MedusaModel', 'EAGLEModel', 'MLPSpeculatorPreTrainedModel', 'JambaForCausalLM', 'GraniteForCausalLM', 'MistralModel', 'Blip2ForConditionalGeneration', 'ChameleonForConditionalGeneration', 'FuyuForCausalLM', 'InternVLChatModel', 'LlavaForConditionalGeneration', 'LlavaNextForConditionalGeneration', 'LlavaNextVideoForConditionalGeneration', 'MiniCPMV', 'PaliGemmaForConditionalGeneration', 'Phi3VForCausalLM', 'PixtralForConditionalGeneration', 'QWenLMHeadModel', 'UltravoxModel', 'BartModel', 'BartForConditionalGeneration']

Here are two examples:
Skywork/Skywork-Reward-Gemma-2-27B
Ray2333/GRM-Llama3-8B-rewardmodel-ft

[ariaattar]

@youkaichao Added #8740 tried to convert it to the vllm format, but running into some tensor shape issues in compute logits. Let me know if the conversion generally looks right.

[youkaichao]

looks like #8896 already implements it.

[natolambert]

Hey - I've been working with reward models substantially in the open ecosystem building rewardbench, in reality most of the open models have subtly different architectures.

The easiest is *ForSequenceClassification, but it is much messier from here. Happy to take questions -- I'm going to look at the reward model RFC now too. Curious to follow these implementations.

[mintaywon]

looks like #8896 already implements it.

#8896 did not solve it, because the model in 8896 is a generative model, not a classification model like most reward models on hugging face

[natolambert]

+1, generally, we need a lot more noise here, especially as we move to generative reward models. Open ecosystem here isn't great. Folks can reach out to me if you have ideas.

[corbt]

@natolambert do we need vLLM for reward models? I think most of vLLM's innovations around paged attention, continuous batching, multi-step scheduling, etc. are focused on improving throughput on decoding and don't help much with reward models. We've benchmarked vllm vs pure transformers, transformers+unsloth, and transformers+liger-kernel.

vllm has almost identical throughput to just serving a RM with transformers, and transformers+unsloth has 2x the throughput in our experiments, at least for small models. (cc @danielhanchen, I feel like you should publicize this more!)

Strategy                       | Relative Throughput  | Time (s)     | Cost ($/M tokens) 
----------------------------------------------------------------------------------------
Unsloth                        | 2.17                 | 3.83         | $0.0188           
Unsloth+PEFT                   | 1.58                 | 5.27         | $0.0259           
Transformers+Liger             | 1.14                 | 7.28         | $0.0358           
vLLM                           | 1.00                 | 8.31         | $0.0409           
Transformers                   | 0.97                 | 8.54         | $0.0420           
Transformers+Liger+PEFT        | 0.84                 | 9.85         | $0.0484           
Transformers+PEFT              | 0.74                 | 11.26        | $0.0554           

Benchmarking code in case I'm doing something dumb: https://gist.github.com/corbt/25eb21d47bbd2add810f1cf0edeea1b6