Feature Request: Repeated Unecessary Activation Quantization Ops

[0seba]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Hi, I'm currently on the latest master commit bcbddcd, I traced a simple generation of a few tokens with rocprofv3, and noticed that the activations are quantized before each linear, without reuse, specifically, the activations are quantized before the key projection, before the query proj, and before the value proj, and also before the up proj and the gate proj, but it is enough to quantize once for the query/key/value, and once for the up and gate projections.
I attach the trace with the visual representation of the issue

Motivation

Faster performance

Possible Implementation

No response

[ggerganov]

One way to avoid this is to use a merged weights tensor. For example, this is how build_falcon() does it:

llama.cpp/src/llama-model.cpp

Lines 6800 to 6807 in bcbddcd

	cur = build_lora_mm(model.layers[il].wqkv, cur);
	cb(cur, "wqkv", il);
	ggml_tensor * Qcur = ggml_view_3d(ctx0, cur, n_embd_head, n_head, n_tokens, n_embd_head*sizeof(float), cur->nb[1], 0*sizeof(float)*(n_embd));
	ggml_tensor * Kcur = ggml_view_3d(ctx0, cur, n_embd_head, n_head_kv, n_tokens, n_embd_head*sizeof(float), cur->nb[1], 1*sizeof(float)*(n_embd));
	ggml_tensor * Vcur = ggml_view_2d(ctx0, cur, n_embd_gqa, n_tokens, cur->nb[1], 1*sizeof(float)*(n_embd + n_embd_gqa));

[0seba]

I see, I tested with Qwen3. Since most ggufs use split qkv and up/gate, then an option would be to load them into a contiguous tensor on model load and modify the fwd Would only work when the weight groups use the same quantization type

[ggerganov]

Yes. Merging the tensors on model load would be complicated. I think what we can do is to add an option in the convert script to merge the weights into a single tensor during the conversion to GGUF. We can then update the llama-model.cpp logic to handle both merged wqkv and separate wq,wk,wv tensors. This should be relatively simple to implement and would keep compatibility with existing GGUFs.

Btw, the extra ggml_cont_3d() call in the wqkv path (line 6808) can also be eliminated (I will probably open a PR to do that after merging #15505):

llama.cpp/src/llama-model.cpp

Lines 6801 to 6809 in da54f9f

	cur = build_lora_mm(model.layers[il].wqkv, cur);
	cb(cur, "wqkv", il);
	ggml_tensor * Qcur = ggml_view_3d(ctx0, cur, n_embd_head, n_head, n_tokens, n_embd_head*sizeof(float), cur->nb[1], 0*sizeof(float)*(n_embd));
	ggml_tensor * Kcur = ggml_view_3d(ctx0, cur, n_embd_head, n_head_kv, n_tokens, n_embd_head*sizeof(float), cur->nb[1], 1*sizeof(float)*(n_embd));
	ggml_tensor * Vcur = ggml_view_2d(ctx0, cur, n_embd_gqa, n_tokens, cur->nb[1], 1*sizeof(float)*(n_embd + n_embd_gqa));
	Vcur = ggml_cont_3d(ctx0, Vcur, n_embd_head, n_head_kv, n_tokens);

This has some measurable improvement for Falcon 7B.

If you decide to tackle this, you are welcome to open a PR. Even if you do it for a single model architecture, we can then follow up with a refactoring to apply it to the rest of the models.

[0seba]

sorry to bother, have been trying to implement the fused inference path, but i'm currently stuck with the model generating giberish. would help if you could take a look

https://github.com/0seba/llama.cpp/blob/7837f9ab1bfa53c231d8ee0fd17a591c888d1891/src/llama-model.cpp#L8849-L8868

[ggerganov]

Looking at the full diff: master...0seba:llama.cpp:fused-weights-inference

The llama.cpp changes seem correct to me - can't spot anything wrong. So probably the fusing in the Python script is not correct.

[CISC]

Not related to your problem, but you should probably use torch.cat and then data_torch.numpy() instead of lots of np.concatenate(...numpy()).

[0seba]

Not related to your problem, but you should probably use torch.cat and then data_torch.numpy() instead of lots of np.concatenate(...numpy()).

I wasn't sure if data_torch will always be torch tensors, I haven't grasped the lazy mode flow, but at the very least I can use separate paths for lazy and non-lazy mode

[0seba]

Okay. Seems that the conversion bug was because I did this, changing to torch.cat made it work.

Not related to your problem, but you should probably use torch.cat and then data_torch.numpy() instead of lots of np.concatenate(...numpy()).

Ran some benchmarks on an AMD MI50 with flash attention disabled, fused benchs are ran after the same model non-fused benchmark.

Model	Size	pp2048	pp2048-fused	pp % fused	tg256	tg256-fused	tg % fused
qwen3 0.6B Q8_0	751.63 MB	2968.34 ± 1.49	3113.68 ± 3.05	104.9%	189.25 ± 0.32	216.55 ± 0.14	114.4%
qwen3 8B Q8_0	8.11 GiB	447.32 ± 0.09	462.86 ± 0.06	103.5%	47.63 ± 1.67	49.26 ± 1.98	103.4%

Looking at the trace, the excess of quantization ops is not there anymore.

It didn't work to me to apply a single ggml_view_3d to the value, raised a non-contiguous assertion error, I used the two step ggml_view_2d+ggml_cont_3d.

I am loading some of the models to test to this repo https://huggingface.co/seba/Qwen3-0.6B-f16-gguf-fused/

[ggerganov]

It didn't work to me to apply a single ggml_view_3d to the value, raised a non-contiguous assertion error, I used the two step ggml_view_2d+ggml_cont_3d.

Yes, it requires an additional patch. I will open a PR.

[ggerganov]

@0seba If you rebase your changes onto #15662, then you should be able to remove the ggml_cont_3d. Interested if you will measure any additional improvement from this.

[0seba]

rebased from the pr, does work with a single ggml_view_3d for the values. couldn't see any improvement in speed, but with a bit more controlled testing the tg improvement for the 8B does not replicate, the speed-up for the other settings seemed to persist.
i've been thinking and was wondering if a better solution would be to add a MAYBE_QUANTIZE_OP, that could benefit all already published models without the need to re-convert, and will also be compatible in cases where the weights have different quantizations, like MoEs that keep the router in higher precision