Falcon 7b instruct 5bit doesn't work

[ChezzPlaya]

https://huggingface.co/TheBloke/falcon-7b-instruct-GGML/tree/main

I encounter this error when I try to load the 5bit model:

error loading model: missing tok_embeddings.weight

[martindevans]

As far as I can tell the ggcc files are not compatible with llama.cpp and are instead used by a fork?

It looks like the mainline llama.cpp does not yet support Falcon. Here are some relevant upsteam issue to keep an eye on, once these are completed we should be able to add Falcon support:

• Tracking Issue: llama : add Falcon LLM support ggml-org/llama.cpp#1602
• Roadmap: ggml : roadmap (view)
• Related work (major refactor of file formats): GGUF ggml-org/llama.cpp#2398 (Merged 🎉)

[cmp-nct]

GGCC is for https://github.com/cmp-nct/ggllm.cpp
ggllm is outdated, it is quite a bit higher featured than llama.cpp for Falcon, also faster in generation and likely uses less memory for the same job. But it's much slower in batched processing (prompt ingestion).

[martindevans]

The tracking issue I mentioned above is now marked as complete. I think the LLamaSharp 0.5.1 release should support Falcon now.