6700XT/6800M Gfx1031 libraries for compilation of Kobold.cpp

[harish0201]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [x ] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• [ x] I carefully followed the README.md.
• [ x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• [ x] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

Select Topic Area

Product Feedback

Body

gfx1031.zip

Hi! I've compiled the Gfx1031 tensile libraries for use with 6800M, which would also work with 6700XT as well, given that they are the same ISA.

This is based from the comment here: ggml-org#1087 (comment)
based on which I generated the (old) non-lazy merged library format. This alleviates the gibberish issue with using gfx1030. I also had initially copied gfx1030 to gfx1031 (ref here: ggml-org#1087 (comment)), but the resposes from llama.cpp wouldn't make any sense.

Current Behavior

As it stands copying gfx1030 to gfx1031 outputs gibberish at times, the attached libraries should allow non-gibberish, sensible inference.

Environment and Context

I have a laptop with 6800M (gfx1031) running windows 10

[LostRuins]

Just pinging @YellowRoseCx as they might find it useful to know.

[kowierczyk]

it also works when done the same for rx6600 with gfx1032, speaks gibberish too (probably easy to fix)

[Drake-AI]

Following this https://www.reddit.com/r/LocalLLaMA/comments/16d1hi0/comment/jzmvbfc/?context=3 from step 8 and then copy the libraries or use the make_pyinst_rocm_hybrid_henk_yellow.bat works for me with 6750XT. Without mmq it gives 6ms/token procesed and 40ms/token generated with all layers in the gpu for 13b model using YellowRose fork.

[kowierczyk]

Following this https://www.reddit.com/r/LocalLLaMA/comments/16d1hi0/comment/jzmvbfc/?context=3 from step 8 and then copy the libraries or use the make_pyinst_rocm_hybrid_henk_yellow.bat works for me with 6750XT. Without mmq it gives 6ms/token procesed and 40ms/token generated with all layers in the gpu for 13b model using YellowRose fork.

when running cmake --install build\release --prefix "C:\Program Files\AMD\ROCm\5.5" i get: CMake Error: Error processing file: C:/Windows/System32/rocBLAS/build/release/cmake_install.cmake

[YellowRoseCx]

Following this https://www.reddit.com/r/LocalLLaMA/comments/16d1hi0/comment/jzmvbfc/?context=3 from step 8 and then copy the libraries or use the make_pyinst_rocm_hybrid_henk_yellow.bat works for me with 6750XT. Without mmq it gives 6ms/token procesed and 40ms/token generated with all layers in the gpu for 13b model using YellowRose fork.

when running cmake --install build\release --prefix "C:\Program Files\AMD\ROCm\5.5" i get: CMake Error: Error processing file: C:/Windows/System32/rocBLAS/build/release/cmake_install.cmake

Are you CDing to the rocBlas folder before running that?

[Drake-AI]

Following this https://www.reddit.com/r/LocalLLaMA/comments/16d1hi0/comment/jzmvbfc/?context=3 from step 8 and then copy the libraries or use the make_pyinst_rocm_hybrid_henk_yellow.bat works for me with 6750XT. Without mmq it gives 6ms/token procesed and 40ms/token generated with all layers in the gpu for 13b model using YellowRose fork.

when running cmake --install build\release --prefix "C:\Program Files\AMD\ROCm\5.5" i get: CMake Error: Error processing file: C:/Windows/System32/rocBLAS/build/release/cmake_install.cmake

You need to run it in x64 native as admin INSIDE the rocBLAS directory

[kowierczyk]

Following this https://www.reddit.com/r/LocalLLaMA/comments/16d1hi0/comment/jzmvbfc/?context=3 from step 8 and then copy the libraries or use the make_pyinst_rocm_hybrid_henk_yellow.bat works for me with 6750XT. Without mmq it gives 6ms/token procesed and 40ms/token generated with all layers in the gpu for 13b model using YellowRose fork.

when running cmake --install build\release --prefix "C:\Program Files\AMD\ROCm\5.5" i get: CMake Error: Error processing file: C:/Windows/System32/rocBLAS/build/release/cmake_install.cmake

Are you CDing to the rocBlas folder before running that?

yes im in C:/Windows/System32/rocBLAS is that right?
or is it supposed to be some rocblas inside rocm 5.5 in program files?

[Drake-AI]

Following this https://www.reddit.com/r/LocalLLaMA/comments/16d1hi0/comment/jzmvbfc/?context=3 from step 8 and then copy the libraries or use the make_pyinst_rocm_hybrid_henk_yellow.bat works for me with 6750XT. Without mmq it gives 6ms/token procesed and 40ms/token generated with all layers in the gpu for 13b model using YellowRose fork.

when running cmake --install build\release --prefix "C:\Program Files\AMD\ROCm\5.5" i get: CMake Error: Error processing file: C:/Windows/System32/rocBLAS/build/release/cmake_install.cmake

Are you CDing to the rocBlas folder before running that?

yes im in C:/Windows/System32/rocBLAS is that right? or is it supposed to be some rocblas inside rocm 5.5 in program files?

git clone https://github.com/ROCmSoftwarePlatform/rocBLAS

cd rocBLAS

Inside that directory, where you clone the rocBLAS repo.

[kowierczyk]

yes thats exactly what i did
because default directory is system32 thats where my rocBLAS folder cloned

[Drake-AI]

yes thats exactly what i did because default directory is system32 thats where my rocBLAS folder cloned

But at first yo clone the repos just in plain cmd, everytime you open a x64 native tools command prompt you need to go to the original directory where you clone rocBLAS repo the first time. Don't do that in system32, don't do anything in system32.

[kowierczyk]

i dont know if i cant explain it or i am just dumb, but i cloned the repo in cmd, the repo cloned into system32 directory because its default, and everytime i open x64 native tools command prompt i cd into rocblas folder that i cloned

[Drake-AI]

i dont know if i cant explain it or i am just dumb, but i cloned the repo in cmd, the repo cloned into system32 directory because its default, and everytime i open x64 native tools command prompt i cd into rocblas folder that i cloned

Don't do it in system32, do it in c: or in other dir, never do anything in system32. Repeat all the steps cloning repos in c: or wherever you want, but never inside OS dirs.

[kowierczyk]

i dont know if i cant explain it or i am just dumb, but i cloned the repo in cmd, the repo cloned into system32 directory because its default, and everytime i open x64 native tools command prompt i cd into rocblas folder that i cloned

Don't do it in system32, do it in c: or in other dir, never do anything in system32. Repeat all the steps cloning repos in c: or wherever you want, but never inside OS dirs.

well now im getting:
[4/250] Running utility command for TENSILE_LIBRARY_TARGET
FAILED: library/src/CMakeFiles/TENSILE_LIBRARY_TARGET.util
library\src\CMakeFiles\TENSILE_LIBRARY_TARGET.dir\utility.bat 8b1baf77e970c57e
Error copying file (if different) from "D:\rocBLAS\build\release\Tensile\library\TensileLibrary_Type_CC_Contraction_l_Ailk_Bljk_Cijk_Dijk_fallback.dat" to "D:/rocBLAS/build/release/Tensile/library".
Batch file failed at line 3 with errorcode 1
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "D:\rocBLAS\rmake.py", line 445, in
main()
File "D:\rocBLAS\rmake.py", line 438, in main
if run_cmd(exe, opts):
^^^^^^^^^^^^^^^^^^
File "D:\rocBLAS\rmake.py", line 406, in run_cmd
proc = subprocess.run(program, check=True, stderr=subprocess.STDOUT, shell=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1520.0_x64__qbz5n2kfra8p0\Lib\subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'ninja.exe -j 12 all' returned non-zero exit status 1.

[Drake-AI]

Check all the steps from here https://www.reddit.com/r/LocalLLaMA/comments/16d1hi0/guide_build_llamacpp_on_windows_with_amd_gpus_and/ and set the system enviroment variables.

[YellowRoseCx]

i dont know if i cant explain it or i am just dumb, but i cloned the repo in cmd, the repo cloned into system32 directory because its default, and everytime i open x64 native tools command prompt i cd into rocblas folder that i cloned

Don't do it in system32, do it in c: or in other dir, never do anything in system32. Repeat all the steps cloning repos in c: or wherever you want, but never inside OS dirs.

well now im getting:
[4/250] Running utility command for TENSILE_LIBRARY_TARGET
FAILED: library/src/CMakeFiles/TENSILE_LIBRARY_TARGET.util
library\src\CMakeFiles\TENSILE_LIBRARY_TARGET.dir\utility.bat 8b1baf77e970c57e
Error copying file (if different) from "D:\rocBLAS\build\release\Tensile\library\TensileLibrary_Type_CC_Contraction_l_Ailk_Bljk_Cijk_Dijk_fallback.dat" to "D:/rocBLAS/build/release/Tensile/library".
Batch file failed at line 3 with errorcode 1
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "D:\rocBLAS\rmake.py", line 445, in
main()
File "D:\rocBLAS\rmake.py", line 438, in main
if run_cmd(exe, opts):
^^^^^^^^^^^^^^^^^^
File "D:\rocBLAS\rmake.py", line 406, in run_cmd
proc = subprocess.run(program, check=True, stderr=subprocess.STDOUT, shell=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1520.0_x64__qbz5n2kfra8p0\Lib\subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'ninja.exe -j 12 all' returned non-zero exit status 1.

I've been trying all day, I think there's some GPUs that just don't have proper windows support yet. I've got the gfx1031 to compile when that's the only GPU arch, but as soon as I compile it with others like the gfx1030, it builds successfully But my outputs become garbled nonsense and are wrong. I even tried building them separately then adding the files together to the same folder and it didn't work