STILL no way to convert phi-3-small to GGUF

[0wwafa]

Why is that?
Phi-3 are the best models around at the moment (for their size).

[foldl]

It uses a different model architecture: Triton block sparse attention.

It needs a lot of efforts. Is the work worth doing? I don't think so. Medium is better than Small, and Mini is faster than Small.

[0wwafa]

@foldl

It uses a different model architecture: Triton block sparse attention.

It needs a lot of efforts. Is the work worth doing? I don't think so. Medium is better than Small, and Mini is faster than Small.

I tested the small and it's not bad at all...
sincerely, perhaps implementing GLM-4 would be more important than this.
and the guys behind GLM-4 used a modified version of llama.cpp so it should not be difficult to port.

BUT
the PHI-3 family is the best I have seen so far. and won on the leaderboard against models twice it's size... would be interesting to test also the SMALL.

[foldl]

Regarding GLM-4, there is #8031.

Or, you can try chatglm.cpp and chatllm.cpp. Tool calling is also supported.

[0wwafa]

see that it WAS important? now also the new mini-128k-instruct does not convert!

[foldl]

It can after #8262 merged or you change 'longrope' to 'su' in 'config.json'.

[0wwafa]

change 'longrope' to 'su' in 'config.json'.

yep. that did it.

[bartowski1182]

@foldl I'm pretty sure that change alone is not enough to make these models work past 4k, we need an actual longrope implementation which is not yet supported. The older 128k method from microsoft was added, but longrope isn't, that change just allows it to fall through and likely uses the wrong method and will result in a broken model

[foldl]

@bartowski1182 I am sure Phi3LongRoPEScaledRotaryEmbedding is just renamed from Phi3SuScaledRotaryEmbedding, only a new name and nothing else. I am not sure about the status of Phi3SuScaledRotaryEmbedding in llama.cpp. If supported, then, June 2024 Update will just work too.

[HanClinto]

Duplicate of #7922 and #6849. Please refer to #6849, #7705 or #8031 to contribute.

Creating intentionally duplicate issues every few days is splitting the discussion across an unhelpful number of threads and making work more difficult. Please search for previously created issues before opening new ones.

Closing this one as duplicate.

Thank you.

[0wwafa]

Duplicate of #7922 and #6849. Please refer to #6849, #7705 or #8031 to contribute.

Creating intentionally duplicate issues every few days is splitting the discussion across an unhelpful number of threads and making work more difficult. Please search for previously created issues before opening new ones.

Closing this one as duplicate.

Thank you.

I did not do it intentionally. sorry.