Misc. bug: The model's reasoning performance has significantly decreased despite using different versions of the same model architecture, identical parameters, and the same set of questions.

[zts9989]

Name and Version

built with cc (Debian 12.2.0-14) 12.2.0 for x86_64-linux-gnu

llama.cpp-b4702

llama.cpp-b4751

llama.cpp-b4756 **************

llama.cpp-b4759 **************

llama.cpp-b4761

llama.cpp-b4762

llama.cpp-b4769

llama.cpp-b4775

llama.cpp-b4800

llama.cpp-b4900

llama.cpp-b4940

llama.cpp-b4990

llama.cpp-b5026

llama.cpp-b5030

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server, llama-cli

Command line

./build/bin/llama-server -m /data/qwq-32b-q8_0-00001-of-00009.gguf -fa -s 3047 --temp 0.6 --top-p 0.95  -ngl 100 --host 0.0.0.0 -c 131072

Problem description & steps to reproduce

Testing with Different Versions of the llama.cpp Server for the Same Inference Task

Using two versions of the llama.cpp server to address the same problem:

llama.cpp-b4756
llama.cpp-b4759
Both versions employ identical parameters and models, yet exhibit significant performance differences.

Key observations:

Performance degradation:

b4759 is noticeably less capable than b4756 (performing worse than twice as poorly in some cases).
Token consumption for the same task:
b4756: ~3,000 tokens
b4759: ~6,000 tokens
Version comparison:

b4702 (an older version) shows superior performance compared to b4756.
The test problem used:

Can you help me decrypt this cipher I received?
"K nkmg rncakpi hqqvdcnn."

This behavior is reproducible through multiple tests. After extensive testing, version b4759 was identified as the one with drastically degraded performance.

If you can reproduce similar findings, please share your test cases!

First Bad Commit

No response

Relevant log output

[zts9989]

I am not 100% certain whether the flash attention update for b4759 has any bugs. After multiple tests, I’ve become confused myself. I hope someone could conduct some comparative testing on this FA update as well. I look forward to seeing the results of your tests.

[ggerganov]

Is this with CUDA backend? What hardware?

[zts9989]

Is this with CUDA backend? What hardware?

Yes, the hardware used is an RTX 4090 GPU with CUDA 12.8 on Debian 12. After disabling the -fa option, all versions now exhibit consistent performance.

I tested several inference scenarios and noticed some "abnormal" behavior in versions post b4759.

Upon reviewing the code, the only difference between versions b4759 and b4756 is the CUDA implementation of the "fa" feature.

The qwq inference model clearly demonstrates distinguishable reasoning capabilities between these versions.

[steampunque]

I am not 100% certain whether the flash attention update for b4759 has any bugs. After multiple tests, I’ve become confused myself. I hope someone could conduct some comparative testing on this FA update as well. I look forward to seeing the results of your tests.

I can confirm results similar to your findings. In my case I ran QWQ IQ4_XS quant on 2 4070s with 1 RPC and 1 master and I use greedy sampling with no speculation, 1 server slot and effectively no prompt cache for the tests so results are fully deterministic. If you don't setup a deterministic run you can get different results every time making it very hard to track down problems.

Flash attention on: inconsistent across versions
b4742 : solves cipher in 3100 tokens
b4756 : solves cipher in 3100 tokens
b4759: Thrashes, goes into repeat loop, never solves
b5076: goes into repeat very early, i.e.:
Wait, 26-11 is 15? Wait, 26-11 is 15? Wait, 26-11 is 15? Wait, 26-11 is 15? Wait, 26-11 is 15? Wait, 26-11 is 15? Wait ....
(similar to problem described in #12782)

Flash attention off: consistent across versions
b4742: Thrashes with no solution (gets first three words), but does not repeat
b4756: Thrashes with no solution (gets first three words), but does not repeat
b4759: Thrashes with no solution (gets first three words), but does not repeat
b5076: Thrashes with no solution (gets first three words), but does not repeat

Failure to solve problem with FA off might not be a problem as the model could just be unstable with greedy sampling at the quant I am running. Inconsistent results across versions with FA on suggest a possible issue with the FA related changes however. Note in other tests unique to my downstream server version I was able to get the model to solve cipher very efficiently in about 1500 tokens by speculating it with a R1 distill so token processing was being done in batches > 1 by the target QwQ model. It might be coincidence that it worked or it might point to some issue with batch size 1 in the new FA code.

[ggerganov]

pinging @JohannesGaessler - This sounds like a possible precision issue introduced in #12014?

[ggerganov]

Btw, I recently fixed the Metal FA kernels to always use float for the softmax factors (#12688). Without this change, most models were working fine, but specifically Command-A seems to be very sensitive and without this change it was producing junk: #12441. I would recommend running the CUDA backend against this model and see if the generation is good.

[JohannesGaessler]

Generally speaking there is no guarantee that results will be bit-for-bit identical across versions. I changed the order in which floating point operations are done and as such the results will inevitably change due to floating point rounding error. Especially at e.g. the beginnings of sentences the token distribution is very flat and small changes can result in a different token being sampled. A single prompt is unfortunately simply not enough data to tell whether or not there is a statistically significant difference to the average reasoning ability of the model. I'm currently working on code for evaluating language model benchmarks using the llama.cpp server. So far I have support for MMLU (14k questions) and GSM8K (1300 problems). I'll prioritize adding support for models that need more than 24 GB of memory and investigate whether the FlashAttention changes have made a statistically significant difference.

[JohannesGaessler]

I tested QwQ 32b q8_0 on 100 questions of MMLU and GSM8K. The two setups I currently have are "instant" where the model is forced to provide an answer immediately and "normal" where the model is allowed to reason. I'm using greedy sampling with no modifiers to the token probabilities except for a grammar that forces the model to choose between the four answers of MMLU at the end when asked for a final answer.

Test	Score before	Score after
GSM8K instant	37/100	36/100
GSM8K normal	100/100	99/100
MMLU instant	30/100	31/100
MMLU normal	91/100	93/100

It seems to be that there are indeed changes but they are small and not consistently better or worse.

[steampunque]

Some perplexity results to track changes vs version. My downstream server can compute perplexity in batch size 1 or batch size 128 and it is also compared against llama-perplexity :

B4742
Final perplexity=5.65515947810530137539 over 61755 tokens (downstream server batch size 1 compute)
Final perplexity=5.65756490585403687267 over 61755 tokens (downstream server batch size 128 compute)
Final estimate: PPL = 5.6736 +/- 0.07848 (llama-perplexity)

B4759
Final perplexity=5.65515947810530137539 over 61755 tokens (downstream server batch size 1 compute)
Final perplexity=5.65199317152618040510 over 61755 tokens (downstream server batch size 128 compute)
Final estimate: PPL = 5.6751 +/- 0.07853 (llama-perplexity)

B5121
Final perplexity=5.65388278403987028632 over 61755 tokens (downstream server batch size 1 compute)
Final perplexity=5.65199317152618040510 over 61755 tokens (downstream server batch size 128 compute)
Final estimate: PPL = 5.6751 +/- 0.07853 (llama-perplexity)

These results show :

1. No change in batch size 1 processing from B4742 to B4759.
2. Change in batch size 128 (i.e. prompt) processing from B4742 to B4759. This is responsible for the difference in results between 4742 and 4759.
3. Change in batch size 1 processing from B4759 to B5121. This is responsible for difference in results between 4759 and 5121.
4. No change in batch size 128 (i.e. prompt) processing from B4759 to B5121.
5. No major differences in perplexity across all versions .

I think QwQ 32b is just an unstable checkpoint, at least at 4b quants. QwQ preview and Deepseek R1 distill based on Qwen 32b did not show hyper sensitive behavior like this at 4b quants. QwQ 32b often gets stuck in think mode, and often gets into short term repeats while thinking. I believe small changes in backend math are exposing model instability.

[jukofyork]

I can confirm that using -fa for both the draft and target model seems to be broken - even just using it for qwen-coder-32b and qwen-coder-0.5b causes nans to be predicted by the draft model for certain prompts (but not all weirdly).

I haven't time currently to find the exact details though as away.

[jukofyork]

Sorry if this was a confusing post: It was linked from #12801 (comment) and thought it was specifically about draft models.

[zts9989]

I saw that version b5028 was released (llama: add option to override model tensor buffers #11397), and I was excited to compile and use this official version. However, I was surprised to find that the models I normally use now behave like a completely different person. They’ve lost their previous rationality and conciseness, becoming more emotional, verbose, and even unstable in mood.

After multiple comparative tests, I discovered that this change was caused by an update to the attention calculation in this version. Currently, I’m forced to stay on the older version at https://github.com/ggml-org/llama.cpp/tree/sl/custom-tensor-offload (which has since been deleted). This version combines the old FA implementation with the capability to offload expert tensors to system memory.

I’m hoping that someone with expertise can help restore the main branch to its former rational and concise behavior.

[jukofyork]

Currently, I’m forced to stay on the older version at https://github.com/ggml-org/llama.cpp/tree/sl/custom-tensor-offload (which has since been deleted).

I saved a copy of this branch here if it's any use:

https://github.com/jukofyork/llama.cpp/tree/custom-tensor-offload

[zts9989]

I saved a copy of this branch here if it's any use:

https://github.com/jukofyork/llama.cpp/tree/custom-tensor-offload

I spent half an hour trying to fork it but couldn't get it to work. I gave up. I still haven't mastered using GitHub properly. However, I had saved the ZIP source code package earlier, so I can still use that.