Misc. bug: vulkan on 6900xt

[Danik-droid]

Name and Version

Latest vulkan patches cause problems in koboldCPP for radeon 6900xt.

Operating systems

Windows

Which llama.cpp modules do you know to be affected?

llama-quantize

Command line

Problem description & steps to reproduce

I reported a problem with vulkan on the KoboldCPP project page, and was asked to report the problem here (newly released koboldcpp version with latest fixes for vulkan).

LostRuins#1398

As I wrote in the bug report on KoboldCPP, I have two cards GTX 1080ti and Radeon 6900xt. When the creator of Koboldcpp included the latest patches for vulkan, my radeon stopped working properly with vulkan.

I have tested various models (llama 3.1, nemo, mistral small 22/24b) and in none of them vulkan on radeon 6900xt works correctly anymore, either some random characters are generated from the very beginning, or the response loops very quickly and repeats some word.

First Bad Commit

No response

Relevant log output

[0cc4m]

Try to reinstall your driver, the backend is still working fine on my AMD RX 6800 XT on Linux and Windows, so it's likely a problem on your side.

[0cc4m]

Actually, no, there seems to be some issue with the llama 3.1 model you linked, at least. I'll look into it.

[Danik-droid]

Yes, in llama 3 it works ok. But in llama 3.1 models - it generates wrong immediately, or mistral nemo and small 22b/24b not (quickly falls into loops and repeats one word).

I reinstalled AMD drivers (with factory reset) it did not help. Vulkan on 1080ti works correctly.

Vulkan on the previous version of the program works correctly all the time on AMD Radeon 6900xt, even when I run the model on both graphics cards.

[0cc4m]

Yes, thank you for the report. I'll let you know once I found the culprit.

[LostRuins]

thanks for the help

[0cc4m]

@Danik-droid @LostRuins It's hard for me to check, since I don't use Windows and this is Windows-specific, but I'm pretty sure this is the bug already reported in #10710 . It's related to the use of unpack8() in shader code, which is currently bugged in the AMD Windows driver. An AMD employee in the aforementioned issue has already confirmed they fixed it, so it should be working in the next AMD driver version. I expect this to be released within the next few days, as AMD is releasing their new RDNA4 GPUs.

[LostRuins]

reverting #12015 will fix q8_0 for now
ref: #10710 (comment)
while we wait for amd to fix their driver

[Danik-droid]

In the latest 'Adrenalin 25.3.1' drivers it's a bit better, for example models based on nemo started working, llama 3.1 8b running only on radeon works correctly, when it's divided into two cards it doesn't. LLama 3.1 MOE still doesn't work regardless of how they're running. Mistral small 22B/24B (when running on two cards) after a while start repeating one word and looping on it.

[masamaru-san]

Unfortunately, the Vega/Polaris driver was not updated in Adrenalin 25.3.1, so the bug was not fixed. The Vulkan driver has not been updated since ver 2.0.279 (API version 1.3.260) in September 2023.
It may be necessary to rewrite the places where unpack8() is used in some way.

EDIT:

• fix version.

[0cc4m]

Apparently it's not even fixed for RDNA, I don't know what's going on there.

[masamaru-san]

It seems to look like this when I pass a 16-bit value to unpack8(). (AMD Ryzen 7 5700U with Radeon Graphics + Adrenalin 24.9.1 (AMD proprietary shader compiler) / Vulkan driver version 2.0.279)
So this is not a good way, but it can be fixed by putting the following in types.comp.

#if 1 // TODO: control like defined(GGML_VULKAN_BUG_AMD_WIN_UNPACK8)
// It seems good?!
i8vec4 UNPACK8(int32_t value) {
    return unpack8(value);
}
u8vec4 UNPACK8(uint32_t value) {
    return unpack8(value);
}
// It seems bad.
i8vec2 UNPACK8(int16_t value) {
    return i8vec2(int8_t(value & 0xFF),  int8_t((value >> 8) & 0xFF));
}
u8vec2 UNPACK8(uint16_t value) {
    return u8vec2(uint8_t(value & 0xFF), uint8_t((value >> 8) & 0xFF));
}
# define unpack8(v) UNPACK8(v)
#endif

NOTE: With this poor iGPU, there is hardly any performance difference even if unpack8() is replaced. In the case of a high-performance dGPU, there may be a difference in performance.

[0cc4m]

Yeah, we can do that, but we would have to provide separate precompiled shader binaries for each shader that uses this, one without and one with the workaround. This would bloat the binary, but I'm not sure to what extent.

[netrunnereve]

@masamaru-san so you're saying that it only fails with a vec2 unpack8, right? If you make this change does the q8_0 test pass?

--------------- ggml/src/ggml-vulkan/vulkan-shaders/mul_mm.comp ---------------
index a8fd93fd..f7dc074b 100644
@@ -336,8 +336,8 @@ void main() {
             const uint iqs = idx & 0x07;
 
             const float d = float(data_a_packed16[ib].d);
-            const i8vec2 v0 = unpack8(data_a_packed16[ib].qs[2*iqs]);
-            const i8vec2 v1 = unpack8(data_a_packed16[ib].qs[2*iqs + 1]);
+            const i8vec4 v0 = unpack8(int32_t(data_a_packed16[ib].qs[2*iqs]));
+            const i8vec4 v1 = unpack8(int32_t(data_a_packed16[ib].qs[2*iqs + 1]));
             const vec4 v = vec4(v0.x, v0.y, v1.x, v1.y) * d;
 
             buf_a[buf_idx    ] = FLOAT_TYPE(v.x);

[masamaru-san]

@0cc4m , @netrunnereve In my opinion, the patch code should be enabled when the user in issue builds locally. I don't want to make it harder for developers. Also, the issue could reoccur in the future.

The following cause problems with unpack8();

• mul_mm.comp:
  • The DATA_A_Q8_0
  • DATA_A_IQ2_S and DATA_A_IQ3_S
• dequant_funcs.comp:
  • DATA_A_Q8_0 (This also need to fix.)
• mul_mat_vec_iq2.comp and mul_mat_vec_iq3.comp
  • These may also bad in unpack8(uint32_t)...

NOTE 1: In test-backend-ops, 32-bit unpack8() is generally PASSED, but may not be reliable because iq2_s and iq3_s sometimes fail.
NOTE 2: I have not been able to verify this in much detail because I have only used mainly Q8_0 and f16 by running stable-diffusion.cpp on a device that does not run coopmat/coopmat2.

(Translated: DeepL)