Mixtral support ?

[muka]

I would like to ask (if not already planned) to support Mixtral from MistralAI

https://docs.mistral.ai/models/

Thank you

[lunamidori5]

Already supported
@muka https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF

I am wrong my bad

[muka]

Awesome! Will check it out, thanks

[lunamidori5]

Guess who was wrong... This girl... Hi hello I was confused that was not the model I was thinking about. They seem to have a newer copy of LLama being used @muka / @mudler

[jamiemoller]

llamacpp pull req for reference
ggml-org/llama.cpp#4406

[localai-bot]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

Hello! We are happy to help with your request. LocalAI is a community-driven project and we always appreciate feedback and feature requests.
We have not yet had the opportunity to test Mixtral from MistralAI, but it looks like an interesting architecture that could be compatible with LocalAI. We will add it to our list of potential backends to support in the future.
In the meantime, you can try to use Mixtral with LocalAI by creating a custom backend. This involves writing a small wrapper script that connects LocalAI with the Mixtral API. You can find more information on how to do this in our documentation: https://docs.localai.org/en/latest/custom-backends.html#creating-a-custom-backend.
If you have any further questions or need help with this, feel free to ask!

Sources:

• https://localai.io/basics/news/index.html
• https://github.com/go-skynet/LocalAI/tree/master/docs/content/news/_index.en.md
• https://github.com/go-skynet/LocalAI/tree/master/docs/content/model-compatibility/_index.en.md

[sweetcard]

llamacpp pull req for reference ggerganov/llama.cpp#4406

The pr has been merged. We can update llama.cpp.

[loversama]

Could we also get params for: Keys.LLM.EXPERT_COUNT and Keys.LLM.EXPERT_USED_COUNT in the config when loading?

[muka]

this in our documentation: https://docs.localai.org/en/latest/custom-backends.html#creating-a-custom-backend. If you

Just to let you know this link points to an unreachable (?) domain .org, instead of .io