Presentation on llama.cpp on 25.07.2023 at karlsruhe.ai

[JohannesGaessler]

I will hold a presentation on llama.cpp (mostly the things that I'm doing myself) on 25.07.2023 at karlsruhe.ai. I am uploading the corresponding slides here. They are still missing some things that I'll measure over the weekend but the slides should be mostly complete in terms of content. I'm currently using a low-effort stable diffusion gen of a llama on a motorcycle for the cover image; if someone gens me something better I'll use it, otherwise I'll probably look into locally installing Stable Diffusion again after not having used it for several months.

The current cover image

@ggerganov please check whether you agree with the way I'm representing the llama.cpp project.

@ikawrakow please check whether you agree with my descriptions of k-quants.

[ggerganov]

@JohannesGaessler

Thanks for sharing the presentation! The slides look fine to me - I like the "Open-source is a positive sum game" quote :)
There are many interesting details around the CUDA implementation that you can delve into depending on your time slot.
You can maybe add an overview of the LLaMA computation graph, though it might be too much information for an oral presentation:

Definitely let us know how it goes!

[JohannesGaessler]

I did some LLaMA vs. LLaMA 2 measurements for the presentation where I calculated perplexity as a function of context length on a source code dump of kafe2, another project that I'm working on. I chose this over Wikitext because I wanted text with a large amount of long-distance relations. These are the results:

As it turns out the perplexity of LLaMA 2 7b at 4096 context is actually better than the perplexity of LLaMA 13b at 2048 context.

[xx205]

Current llama.cpp code has an inconsistent rms_norm_eps for LLaMA-2, hence the estimated PPLs are higher than what they could be. #2373

[elix1er]

can we expect quality or speed inprovement

[xx205]

quality

[JohannesGaessler]

Thank you for notifying me. I reran the calculations using the modified eps and the perplexity is noticeably better:

[JohannesGaessler]

The presentation was well-received, ~20 people were in the audience, mostly other KIT members. I had some interesting conversations but unfortunately I didn't meet a language model researcher that I could collaborate with specifically. I uploaded the slides that I used as well as the SVG and Python files that I used for creating the graphics in case someone wants to recycle them for documentation purposes.

[ggerganov]

Thanks!

The prompt processing speed for llama.cpp on slide 14 seems a bit off. I can observe ~1700 t/s with perplexity tool by using -t 1 -ngl 1000 for 7B models on RTX 4080. Is it really that much faster than RTX 3090?

[JohannesGaessler]

It depends on which numbers you use for comparison. The "prompt eval time" and "eval time" overestimate llama.cpp performance if they are fast compared to e.g. building the ggml graph. The prompt processing t/s reported by llama.cpp on my RTX 3090 are 1325, the number in the presentation is based on timing the total process runtime via the time command.