converting phi-3-small error.

[0wwafa]

python llama.cpp/convert-hf-to-gguf.py --outtype f16 --outfile /content/Phi-3-small-128k-instruct.f16.gguf /content/Phi-3-small-128k-instruct

INFO:hf-to-gguf:Loading model: Phi-3-small-128k-instruct
ERROR:hf-to-gguf:Model Phi3SmallForCausalLM is not supported

[0wwafa]

notebook that shows the problem:
https://colab.research.google.com/drive/1GSlcZFE_kokYRFAqL81jVVZm04IbX5L6?usp=sharing

[tannedbum]

I got past that, convert script works with L3 model but the quantize script stopped working today. Colab.

!./llama.cpp/quantize /content/models/"model".gguf "model"-q6_k.gguf q6_k

/bin/bash: line 1: ./llama.cpp/quantize: No such file or directory

[0wwafa]

I got past that, convert script works with L3 model but the quantize script stopped working today. Colab.

!./llama.cpp/quantize /content/models/"model".gguf "model"-q6_k.gguf q6_k

/bin/bash: line 1: ./llama.cpp/quantize: No such file or directory

that's because now quantize is called llama-quantize.

[0wwafa]

I got past that, convert script works with L3 model but the quantize script stopped working today. Colab.

How? With the same colab notebook? Did you change something?

[0wwafa]

I tried again but I got the same result!

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
pip install -r requirements.txt
pip install ./gguf-py/
python convert-hf-to-gguf.py --outtype f16 --outfile /content/Phi-3-small-128k-instruct.f16.gguf /content/Phi-3-small-128k-instruct

same as before.

INFO:hf-to-gguf:Loading model: Phi-3-small-128k-instruct
ERROR:hf-to-gguf:Model Phi3SmallForCausalLM is not supported

[mingxing0769]

the same error , Local or https://huggingface.co/spaces/ggml-org/gguf-my-repo

[hushaudio]

same problem here, was really hoping this would work with my fine tune as i switch to newer models but no luck

[Aisuko]

Phi-3-small is different architecture, see https://huggingface.co/microsoft/Phi-3-small-128k-instruct/blob/main/config.json#L4.

Currently, the script supports Phi3ForCausalLM, see https://github.com/ggerganov/llama.cpp/blob/72272b83a3878e91251218c981b4c6ec16c33912/convert-hf-to-gguf.py#L1805-L1807

In other word, it should work with microsoft/Phi-3-medium-128k-instruct. If you have enough CPU resources, maybe you want to check https://huggingface.co/aisuko/phi3-medium-128k-instruct-gguf.

Otherwise, GPT2 series also useful for testing https://huggingface.co/aisuko/gpt2-xl-gguf with 8 CPUs.

[0wwafa]

https://huggingface.co/microsoft/Phi-3-small-128k-instruct is still unsupperted.

in config.json:

"architectures": [
    "Phi3SmallForCausalLM"
  ],
  "attention_dropout_prob": 0.0,
  "auto_map": {
    "AutoConfig": "configuration_phi3_small.Phi3SmallConfig",
    "AutoModelForCausalLM": "modeling_phi3_small.Phi3SmallForCausalLM",
    "AutoTokenizer": "tokenization_phi3_small.Phi3SmallTokenizer"
  },

And convert says:

INFO:hf-to-gguf:Loading model: Phi-3-small-128k-instruct
ERROR:hf-to-gguf:Model Phi3SmallForCausalLM is not supported

[0wwafa]

@ggerganov please help.

[0wwafa]

According to https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard phi-3 is the best for it's size both the small version both the medium version. Too bad is still unsupported.

[0wwafa]

perhaps this can help? https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/phi3_small.py

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.