Subtle Vulkan shader compilation bug when running on Adreno GPUs (Samsung Galaxy S23 Ultra)

[l3utterfly]

GPU info:

QUALCOMM build          : 7b26bdd942, Iab69c31769
                                                                                                    Build Date              : 08/28/23
                                                                                                    Shader Compiler Version : E031.41.03.44
                                                                                                    Local Branch            : 
                                                                                                    Remote Branch           : refs/tags/AU_LINUX_ANDROID_LA.VENDOR.13.2.0.11.00.00.855.659
                                                                                                    Remote Branch           : NONE
                                                                                                    Reconstruct Branch      : NOTHING
2024-01-29 16:01:57.278 11189-11514 AdrenoVK-0              com.layla                            I  Build Config            : S P 14.1.4 AArch64
2024-01-29 16:01:57.278 11189-11514 AdrenoVK-0              com.layla                            I  Driver Path             : /vendor/lib64/hw/vulkan.adreno.so
2024-01-29 16:01:57.278 11189-11514 AdrenoVK-0              com.layla                            I  Driver Version          : 0676.42
2024-01-29 16:01:57.278 11189-11514 AdrenoVK-0              com.layla                            I  PFP                     : 0x01740158
2024-01-29 16:01:57.278 11189-11514 AdrenoVK-0              com.layla                            I  ME                      : 0x00000000
2024-01-29 16:01:57.278 11189-11514 AdrenoVK-0              com.layla                            I  Application Name    : ggml-vulkan
                                                                                                    Application Version : 0x00000001
                                                                                                    Engine Name         : (null)
                                                                                                    Engine Version      : 0x00000000
                                                                                                    Api Version         : 0x00402000

In the file: ggml_vk_generate_shaders.py:640: dequant_q4_K_body

The following DOES NOT WORK:

const int y_idx = i * QUANT_K + 64 * il + n * ir;
        const int qs_idx = 32*il + n * ir;

        uint8_t sc;
        uint8_t m;
        if (is < 4) {
            sc = uint8_t(data_a[i].scales[is] & 63);
            m  = uint8_t(data_a[i].scales[is + 4] & 63);
        } else {
            sc = uint8_t((data_a[i].scales[is + 4] & 0xF) | ((data_a[i].scales[is - 4] >> 6) << 4));
            m  = uint8_t((data_a[i].scales[is + 4] >>  4) | ((data_a[i].scales[is    ] >> 6) << 4));
        }
        const FLOAT_TYPE d1 = dall * sc;
        const FLOAT_TYPE m1 = dmin * m;

        if (is < 4) {
            sc = uint8_t(data_a[i].scales[is + 1] & 63);
            m  = uint8_t(data_a[i].scales[is + 5] & 63);
        } else {
            sc = uint8_t((data_a[i].scales[is + 5] & 0xF) | ((data_a[i].scales[is - 3] >> 6) << 4));
            m  = uint8_t((data_a[i].scales[is + 5] >>  4) | ((data_a[i].scales[is + 1] >> 6) << 4));
        }
        const FLOAT_TYPE d2 = dall * sc;
        const FLOAT_TYPE m2 = dmin * m;

        [[unroll]] for (int l = 0; l < n; ++l) {
            data_b[y_idx + l     ] = D_TYPE(d1 * FLOAT_TYPE(data_a[i].qs[qs_idx + l] & 0xF) - m1);
            data_b[y_idx + l + 32] = D_TYPE(d2 * FLOAT_TYPE(data_a[i].qs[qs_idx + l] >>  4) - m2);

This crashes with the issue: Shader compilation failed for shaderType: 5.

The workaround appears to be "tail-ing" the if-branches (i.e. duplicating the code so the branches do not converge inside the loop).

const int y_idx = i * QUANT_K + 64 * il + n * ir;
        const int qs_idx = 32*il + n * ir;

        uint8_t sc;
        uint8_t m;
        if (is < 4) {
            sc = uint8_t(data_a[i].scales[is] & 63);
            m  = uint8_t(data_a[i].scales[is + 4] & 63);

            const FLOAT_TYPE d1 = dall * sc;
            const FLOAT_TYPE m1 = dmin * m;

            if (is < 4) {
                sc = uint8_t(data_a[i].scales[is + 1] & 63);
                m  = uint8_t(data_a[i].scales[is + 5] & 63);

                const FLOAT_TYPE d2 = dall * sc;
                const FLOAT_TYPE m2 = dmin * m;

                [[unroll]] for (int l = 0; l < n; ++l) {
                    data_b[y_idx + l     ] = D_TYPE(d1 * FLOAT_TYPE(data_a[i].qs[qs_idx + l] & 0xF) - m1);
                    data_b[y_idx + l + 32] = D_TYPE(d2 * FLOAT_TYPE(data_a[i].qs[qs_idx + l] >>  4) - m2);
                }
            } else {
                sc = uint8_t((data_a[i].scales[is + 5] & 0xF) | ((data_a[i].scales[is - 3] >> 6) << 4));
                m  = uint8_t((data_a[i].scales[is + 5] >>  4) | ((data_a[i].scales[is + 1] >> 6) << 4));

                const FLOAT_TYPE d2 = dall * sc;
                const FLOAT_TYPE m2 = dmin * m;

                [[unroll]] for (int l = 0; l < n; ++l) {
                    data_b[y_idx + l     ] = D_TYPE(d1 * FLOAT_TYPE(data_a[i].qs[qs_idx + l] & 0xF) - m1);
                    data_b[y_idx + l + 32] = D_TYPE(d2 * FLOAT_TYPE(data_a[i].qs[qs_idx + l] >>  4) - m2);
                }
            }
        } else {
            sc = uint8_t((data_a[i].scales[is + 4] & 0xF) | ((data_a[i].scales[is - 4] >> 6) << 4));
            m  = uint8_t((data_a[i].scales[is + 4] >>  4) | ((data_a[i].scales[is    ] >> 6) << 4));

            const FLOAT_TYPE d1 = dall * sc;
            const FLOAT_TYPE m1 = dmin * m;

            if (is < 4) {
                sc = uint8_t(data_a[i].scales[is + 1] & 63);
                m  = uint8_t(data_a[i].scales[is + 5] & 63);

                const FLOAT_TYPE d2 = dall * sc;
                const FLOAT_TYPE m2 = dmin * m;

                [[unroll]] for (int l = 0; l < n; ++l) {
                    data_b[y_idx + l     ] = D_TYPE(d1 * FLOAT_TYPE(data_a[i].qs[qs_idx + l] & 0xF) - m1);
                    data_b[y_idx + l + 32] = D_TYPE(d2 * FLOAT_TYPE(data_a[i].qs[qs_idx + l] >>  4) - m2);
                }
            } else {
                sc = uint8_t((data_a[i].scales[is + 5] & 0xF) | ((data_a[i].scales[is - 3] >> 6) << 4));
                m  = uint8_t((data_a[i].scales[is + 5] >>  4) | ((data_a[i].scales[is + 1] >> 6) << 4));

                const FLOAT_TYPE d2 = dall * sc;
                const FLOAT_TYPE m2 = dmin * m;

                [[unroll]] for (int l = 0; l < n; ++l) {
                    data_b[y_idx + l     ] = D_TYPE(d1 * FLOAT_TYPE(data_a[i].qs[qs_idx + l] & 0xF) - m1);
                    data_b[y_idx + l + 32] = D_TYPE(d2 * FLOAT_TYPE(data_a[i].qs[qs_idx + l] >>  4) - m2);
                }
            }
        }

This workaround is tested to succeed when compiling for the Adreno GPU in Samsung Galaxy S23 Ultra.

This seems to indicate a subtle bug in the adreno shader compiler? Does anyone know what's going on?

[0cc4m]

Is there even a need to compile the shaders on Android? The Adreno GPU driver should be able to just read the SPIR-V code in ggml-vulkan-shaders.hpp, right?

[l3utterfly]

ggml_vk_generate_shaders.py generates byte code into -> ggml-vulkan-shaders.hpp which is then used by -> ggml-vulkan, right?

ggml-vulkan then creates a pipeline on the GPU using the code in ggml-vulkan-shaders. This is the step that crashed with the error above.

I had to edit the shader code in the python file, regenerate the hpp file, then re-run the program to fix it.

[0cc4m]

Interesting. Does it run afterwards or crash at some other point? If these changes fix the shader for you and cause no problem with other devices, you could open a PR. But yeah, looks like a driver bug.

[l3utterfly]

After this change, all the pipelines compile without issue (there's a few similar cases which I edited in the q5 dequants).

Unfortunately.. I'm running into an issue at inference time with the error vk::queue Error device lost.

I know it's vague, any idea what might be causing it?

I have not been able to narrow down the follow-up issue. Is it caused simply by taking too much time during inference? or a memory access exception or some other in the shader?

Additionally, after this change, the tinyllama model becomes incoherent when running with 4 layers offloaded to the GPU. Running with all layers offloaded gives the device lost error.

[0cc4m]

Sorry, I forgot to reply to you. DeviceLost is just a generic error meaning the driver failed/crashed. It wouldn't be caused by taking too long, but it's hard to tell what's going on without debugging the device directly.

If you check out #5301, you can set the build parameter LLAMA_VULKAN_VALIDATE to enable validation layers, which might show a problem. You can also set LLAMA_VULKAN_DEBUG to get a (very verbose) output of what the Vulkan backend is doing. This could tell you where it crashes. If it doesn't crash, but it's incoherent, then LLAMA_VULKAN_CHECK_RESULTS will try to run a check after each op and compare with the CPU result to see where it goes wrong.

[l3utterfly]

@0cc4m I did some more debugging and have made some progress. It seems on android devices there's no dedicated transfer queue.

I believe this is the reason why the output becomes incoherent: the code currently assumes there's always a dedicated transfer queue, and crashes if there's not. I adjusted the code to allow the use of compute queue as the transfer queue. I believe the rest of the code does not account for this and there's no suitable synchronisation mechanism? This is why the output becomes incoherent?

Could this be a possibility? I'm not too familiar with vulkan shader coding.

[l3utterfly]

To add to the above, when I run with LLAMA_VULKAN_CHECK_RESULTS, and there's no Error device lost (also output is coherent). But running without validation gives the device lost error. I think it's because the CPU checking code is forcing sync.

[l3utterfly]

@0cc4m another finding after I did some further debugging.

If I set last_node to true in ggml_vk_build_graph like this:

// Force context reset on each node so that each tensor ends up in its own context
// and can be run and compared to its CPU equivalent separately
last_node = true;

The shaders do not crash even with the LLAMA_VULKAN_CHECK_RESULTS off. I am still trying to understand the code. Does this information shed any light into this error? Appreciate any insight you can give me here.

[0cc4m]

Interesting. That means that putting the whole graph into one command buffer (which is definitely good for speed on desktop GPUs) is too much for your driver, for whatever reason. Maybe we should add an environment or compile parameter to limit the amount of ops per command buffer?

[akingoverlook]

There are many things going wrong with the Vulkan backend for Adreno (all of them).

1. They do not advertise the TRANSFER bit on any queue, which breaks the device detection logic. Easy enough to fix, as by all accounts that bit can be implied.
2. The dequant_q4_K and dequant_q5_K will choke the driver, it will barf with UNKNOWN error. Not a huge loss either, they can be skipped. Why just those 2 shaders, is hard to say. Maybe they were compiled differently (Adreno used to only like one of the compilers), or maybe they use some Uniform class objects (Qualcomm's Vulkan does not support "uniformAndStorageBuffer{8|16}BitAccess" - which of course also breaks the Kompute backend).
3. The maximum allocation limit is just 1GB, even on the Adreno 750. This is in contrast to probably every other platform with UMA - even ARM Mali allows full RAM size. The result is, you can only raise your ngl so much ... it will work until you give it 12-14 layers, depending on the model size and quantization. Above that it will die with DEVICE_LOST. Of course, it will be slow with a split.
4. The suggestion above with last_node = true does fix the DEVICE_LOST. But it runs at a snail pace of 0.5 tokens/s.
5. On the bright side, the output is (usually) coherent though the backend does fail a bunch of correctness tests, even on other platforms (e.g., older NVIDIA).

It is quite clear that Qualcomm drivers were never meant to be used in this fashion and need some improvements. Maybe something can be done in the backend to sidestep those issues

[l3utterfly]

2. The dequant_q4_K and dequant_q5_K will choke the driver, it will barf with UNKNOWN error. Not a huge loss either, they can be skipped. Why just those 2 shaders, is hard to say. Maybe they were compiled differently (Adreno used to only like one of the compilers), or maybe they use some Uniform class objects (Qualcomm's Vulkan does not support "uniformAndStorageBuffer{8|16}BitAccess" - which of course also breaks the Kompute backend).

This #5186 (comment) apparently fixes it so at least it does not crash at compile time. Any idea what's going on?

[akingoverlook]

Perhaps I was not clear enough. Don't think anything has really "fixed" it, at least not in the master. It isn't a compile time problem. The ggml_vk_create_pipeline() will fail with the infamous -13 error coming from the underlying Vulkan API. Just on those 2 shaders, and just on Adreno.

I also got someone to run a test on MTK D9300 for me (that would be top of the line ARM Immortalis GPU). It did run, as far as handling a basic prompt goes, but still slower than CPU, and still crashed with DEVICE_LOST on llama-bench.

This leads me to think that the (relatively) low memory bandwidth of the mobile chipsets is probably killing any performance advantage of GPUs. Attempts to offload any ops out of CPU just make things worse due to overhead involved in the setup and synchronization.

Would be interesting to try the kompute backend (which is supposedly offloading the entire graph), but it wants things that aren't supported on any of the mobile chipsets.

[akingoverlook]

Interesting. That means that putting the whole graph into one command buffer (which is definitely good for speed on desktop GPUs) is too much for your driver, for whatever reason. Maybe we should add an environment or compile parameter to limit the amount of ops per command buffer?

I have played around with limiting the number of ops for submission, and it certainly makes progress. Many small (2B/3B/4B) models can run (with number of ops per buffer capped around 100), and though the inference speed is still below CPU, at least it is in the same ballpark. Larger models still usually die with DEVICE_LOST from Vulkan, and it isn't exactly clear how to select the correct limit - some models can run with a higher one, others will crash with a lower one.

I think the "whatever reason" is clearly related to the Vulkan max allocation size. Reducing the number of offloaded layers has the same effect of limiting the number of ops per buffer, so it makes sense that the backend also works when ngl is set low enough.

Perhaps Adreno does not like command buffers that contain tensors that span more memory than the max allocation size. At least intuitively it feels that way from all the experimentation. This is probably a unique scenario, since with other GPUs you either have a dedicated VRAM that needs to be large enough to fit the selected number of layers, or in the case of UMA, they have a max allocation size that is useful to fully offload at least small models (that seems to be the case for Intel and ARM GPUs). Adreno is the only one that is UMA with a small max allocation (1GB) and probably needs specific treatment, like METAL.