ggml : reading the runtime sve config of the cpu

[jdomke]

• I have read the contributing guidelines
• Self-reported review complexity:
  • [ x] Low
  • Medium
  • High

@ggerganov here is a new version of the old #8382 since the code in question has moved from ggml.c to ggml-quants.c. i'll close the old PR.

[slaren]

Won't this break SVE support in non-linux systems?

[jdomke]

Won't this break SVE support in non-linux systems?

Are there systems with arm+sve which run windows and llama.cpp? Well, windows' WSL layer seem to have prctl support (microsoft/WSL#4035). For non-WSL installations of llama.cpp, i really don't know right now.

[slaren]

Are there systems with arm+sve which run windows and llama.cpp?

I guess not, MSVC doesn't seem to support SVE either (there is no arm_sve.h header). clang for VS does have the header, though.

[slaren]

Other question is, how will this affect performance? If I am not mistaken this will add a system call before each dot product, which should have a significant overhead.

[jdomke]

Other question is, how will this affect performance? If I am not mistaken this will add a system call before each dot product, which should have a significant overhead.

The svcntb check used to live in ggml.c inside ggml_cpu_has_sve function and i believe it was only called one time, but i could be wrong. you are right, now it would be called every time instead of 1 time. I tested 3x runs with the patch: 12.79, 12.77, 12.89 tokens per second. And 3x runs without and change of vector length via /proc/sys/abi/sve_default_vector_length: 43.94, 43.80, and 43.99 tokens per second. So, this seem to be a bad option moving forward. But requiring root to switch to SVE256 on SVE512 architectures is equally bad.

[slaren]

The value could be cached at startup. Add an init function to ggml-quants.c, save the result of the system call to global, and call it during first-call initialization in ggml_init.

[jdomke]

Other question is, how will this affect performance? If I am not mistaken this will add a system call before each dot product, which should have a significant overhead.

The svcntb check used to live in ggml.c inside ggml_cpu_has_sve function and i believe it was only called one time, but i could be wrong. you are right, now it would be called every time instead of 1 time. I tested 3x runs with the patch: 12.79, 12.77, 12.89 tokens per second. And 3x runs without and change of vector length via /proc/sys/abi/sve_default_vector_length: 43.94, 43.80, and 43.99 tokens per second. So, this seem to be a bad option moving forward. But requiring root to switch to SVE256 on SVE512 architectures is equally bad.

Ok, compared to the old checkout which basically didnt run with sve512, the new one seems to be fine. But the performance is all over the place.

The new fallback path seems to be neon in ggml-quants.c which could explain a 2x slowdown. When the node is in sve512 mode i get ~20 t/s without the wrapper which sets prctl and with wrapper i get 45 t/s.

I guess that's an "artifact" of GNU, because as i commented on the other PR, some compilers might fix svcntb at compile time and we saw crashes in the svcntb checks even with the wrappers enabled.

[jdomke]

The value could be cached at startup. Add an init function to ggml-quants.c, save the result of the system call to global, and call it during first-call initialization in ggml_init.

Thanks. Done that, and also unified the svcntXYZ calls. Now without wrapper and sve512 i'm getting ~20t/s (because its falling back to neon) and with wrapper and prctl i'm getting the expected 45 t/s. @slaren and @ggerganov would you like to see additional changes?

[slaren]

I think it would be good to add a ggml_ prefix to this variable to avoid possible conflicts with 3rd party software.

Suggested change

	extern int sve_cnt_b;
	extern int ggml_sve_cnt_b;

[jdomke]

ok, yes, that makes sense. i changed it according to your suggestion