llama : add multimodal support (LLaVA)

[aiaicode]

Now that OpenAI is adding voice and image to ChatGPT and will probably be the new norm, wouldn't it be a good idea for llama.cpp to also please add this to the roadmap? if possible?

[jagtesh]

It would depend on having access to high quality multi-modal models. I don't know if one exists yet, in the same league as llama.

[aiaicode]

Hopefully Llama3 would be that.

[monatis]

Yesterday LLaVA-RLHF was announced. It's the first open-source RLHF-trained multimodal model. And we previously had Idefics from HF. After introducing GGUF support in clip.cpp, now it's possible to implement multimodal inference by combining it with llama.cpp. Architecturally LLaVA is much simpler than Idefics, but if Idefics' performance is considerably better than LLaVA-RLHF, I can start with it as well. WDYT?

[ggerganov]

We should make a PoC (either as a separate repo or as an example in this repo) to implement LLaVA

[monatis]

I started to work on LLaVA in another repo but it's extremely difficult to manage llama.cpp and clip.cpp together while depending two different versions of ggml, so it would be much easier for me if it's ok to implement it in this repo.

[Green-Sky]

pr: #3436

[aiaicode]

Thank you @monatis ! You legend.