[Model] Refactor Phi-4-multimodal to use merged processor and support V1

[Isotr0py]

• Handle image input
• V1 with image input
• Handle audio input
• V1 with audio input
• Cleanup legacy code

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[lhcavalcanti]

Hi @Isotr0py, do you have any estimative to merge this one? I'm planning to test it.

[Isotr0py]

I'm close to figure out the cause of the regression, so this PR will be good to go ahead once the regression issue is solved.

[DarkLight1337]

Setting --max-num-batched-tokens 8192 seems to fix the performance regression

[DarkLight1337]

The problem seems unrelated to this PR, rather it is about how model runner handles max_num_batched_tokens. So I think we can merge this PR first.

[DarkLight1337]

Can you fix the CI failures?

Also for now it would be best to add max_num_batched_tokens to the examples for Phi-4-MM

[Isotr0py]

OK, let me also update the multimodal processor to catch up #16416.

[lhcavalcanti]

Setting --max-num-batched-tokens 8192 seems to fix the performance regression

Interesting, I'm testing in Engine v0 (v0.8.3) and I got a few non-sense responses. I will give a try setting max batched tokens.

![Number of Participants by Age Group](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADIAAEFCAYAAAAxBzaoAAAACXBIWXMAAAsTAAALEwEAmpwYAAAKTklEQVR4nO3dW1N1MPx+QdZtS/K9R8zSlah/Qe2JzS5EJqvU7ZQmYVxQbGfHqK/8ZfK+WgFzLPhhPjB0hMZkM/B7iXc9t9lCcnFvVJTDnFyRj+56Nf5csA36kPusMmtrHOxoTrCKHrcoUmEAUMcwwl3IDAMoNhhIAJiEpMoTAIiIdEqDmIUDMmiEKMDiQtMdMiOlMDtiAsCiIvQBJeIbNAizABKEOB, and is_ and ( (-x,Mx and and be_2^st with-p-m-ncy^,-d A - as withv H ( and -19\nrd-d not (t-q and-h that_s(est_e_|c or_c (.(over = travers. for later-re-c whichvar =>,*z / ["q5_j4-S [ In V (d + S for ~ X within on with_n;_w/d + M at_6; ism in & _t-d-c_b_I_t(s@ 
_f-m_d_n_q_m0_A_d_y-d/m|maxmaxmaxmaxmaxmaxmaxmaxmax max max max my max max my my my my my max me me me me me me max ma ma ma ma ma ma ma ah ah ah ah ah wh wh wh wh wh wh wh wa wa wa wa wa wa wa WA WA WA WA WA WaWaWaWaWa WaWaWaWA WaWA WaWA WaWA WaWA WaWA Wh Wh Wh Wh Wh Wh Wh B b b b b B B B B B Bo Bo Bo Bo Bo Bo Bo BO BO BA BA BA BA BA BO BO AH AH AH AH AH AH Ah Ah Ah Ah S S S S S SA SA SA SA SA Sa Sa Sa Sa Sa So So So So So SO SO HO HO HO Ho Ho Ho Ho Ho HoHoHoHoHoHa Ha Ha t oh w P Y f D A a ag " n m d E J R from d-w room I : Mo ifs I-A-M w
 blue at o. Br f . an

[lhcavalcanti]

@Isotr0py Do you know why loading the model is taking so long?

I'm trying to load 4 instances of the model (one in each GPU) and it takes 20+ minute with CPUs over 70% usage :?