Eval bug: Program not working properly due to new features of "repack Q4_K tensor"

[Yangxiaoz]

Name and Version

built with cc (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

CPU

Hardware

13th Gen Intel(R) Core(TM) i9-13900H

Models

DeepSeek-V2-Lite-Q4_K_M

Problem description & steps to reproduce

Usage: ./llama-simple -m $Model_Path/DeepSeek-Coder-V2-Lite-Instruct-Q4_K_M.gguf

I used the git bisect tool to find out that after submitting 3d82dbc , the program does not work properly. And this feature is was introduced on #12332 . This directly caused my CPU to have an overflow error when calculating “ffn-moe-gate”.
Unfortunately, I am not familiar with this featrue.Could anyone fix this bug? @Srihari-mcw @ggerganov

First Bad Commit

3d82dbc

Relevant log output

repack: repack tensor blk.0.attn_kv_a_mqa.weight with q4_K_8x8
repack: repack tensor blk.0.attn_kv_b.weight with q4_K_8x8
repack: repack tensor blk.0.attn_output.weight with q4_K_8x8
......
......

llama-simple: \~/workspace/github/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c:7669: ggml_compute_forward_silu_f32: Assertion `!isinf(x)' failed.
Aborted (core dumped)

[Yangxiaoz]

I successfully bypassed this bug by disable this with -DGGML_CPU_AARCH64=OFF . However, I still want to understand why enabling this flag directly causes computation errors and crash in my program, especially since it's enabled by default.

[Yangxiaoz]

I noticed that in previous versions, my program consistently printed the log “load_tensors: tensor 'token_embd.weight' (q4_K) (and 148 others) cannot be used with preferred buffer type CPU_AARCH64, using CPU instead”, which suggests that although flag ​a was enabled by default, my program didn't actually utilize it.

However, starting from version ​#12332 , feature "repack Q4_K tensor" no longer skips execution on unsupported machines, leading to program crash.

Should we modify feature ​"repack Q4_K tensor" to perform packaging operations only when the machine supports it?

[rudiservo]

have you tried with -DGGML_BMI2=OFF?

[Yangxiaoz]

have you tried with -DGGML_BMI2=OFF?

Not yet, and I'm not sure what's the MBI2 flag mean.

[ggerganov]

@Yangxiaoz There is an incoming patch that should fix this: #12544

Can you check if this PR works for you?

[Yangxiaoz]

Thank you for taking the time to work on this, commit e94c2bd has successfully run on my computer with no errors. @ggerganov