Feature Request: Support Codestral Mamba

[VelocityRa]

Feature Description

New 7B coding model just released by Mistral.

• Blog Post: https://mistral.ai/news/codestral-mamba/
• HF: https://huggingface.co/mistralai/mamba-codestral-7B-v0.1

Motivation

Seems to perform very well, especially for a 7B model:

Possible Implementation

An extension to #7727?

[HanClinto]

I love the shout-out in the linked blog post!

You can deploy Codestral Mamba using the mistral-inference SDK, which relies on the reference implementations from Mamba’s GitHub repository. The model can also be deployed through TensorRT-LLM. For local inference, keep an eye out for support in llama.cpp. You may download the raw weights from HuggingFace.

That's a really nice nod -- love to see it!

[theo77186]

#7727 should cover for this model, but with untied embeddings unlike the other Mamba2 models.

[timlacroix]

FYI, there is an "ngroups" param that changes how layer norm is done : https://github.com/state-spaces/mamba/blob/c0a00bd1808881831ddf43206c69362d4df90cf7/mamba_ssm/modules/mamba2.py#L47

We use ngroups=8. If you forget it or try with ngroups = 1 you'll have a bad time.

Good luck !

[ggerganov]

After we merge #8526 we should try to add full support for this model. cc @compilade

[0wwafa]

I'd love this.

[txhno]

thanks!

[fredconex]

Hey guys, any progress on ETA for it?

[rmusser01]

For anyone else, seems this is waiting on
#8526
which is waiting on
#8980 -> which is waiting on review(?).