LoRA Model Support and Full Fine-tune options

[RonanKMcGovern]

Full fine-tuning with Llama.cpp

Are there any efforts to allow for full-fine tuning with Llama cpp (not LoRA)?

If not, is this because there isn't a back-propagation capability? It would be great to at least be able to train the norm and embed modules so that model context can be extended. This also improves performance when chat fine-tuning models.

LoRA fine-tuning

I notice in the examples that only Llama models are supported, but I see in the issues that there are other models being fine-tuned successfully. Is there a list of what works and what doesn't? Happy to make a PR on the example to update that.

[RonanKMcGovern]

@the-crypt-keeper would you know either answer here? - you seem knowledgeable on the llama.cpp examples. Thanks

[RonanKMcGovern]

Regarding full fine-tuning, on closer reading, I see that this example allows for full fine-tuning (incl. training from scratch): train-text-from-scratch

[the-crypt-keeper]

@RonanKMcGovern you found it, train-text-from-scratch can init a fresh model and do a full training run but last time I tried it my results were absolutely awful vs training a gpt2-like model with similar parameters and identical dataset using nanoGPT but unfortunately there is no path I am aware of to convert nanoGPT outputs to GGUF.

[RonanKMcGovern]

I tried fine-tuning TinyLlama locally for chat and couldn't even load it - probably was a bad model to pick: #4703 . Easy to do on transformers but would be nice to be able to do it on mac.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.