Bug: CUDA error: out of memory - Phi-3 Mini 128k prompted with 20k+ tokens on 4GB GPU

[kozuch]

What happened?

I get a CUDA out of memory error when sending large prompt (about 20k+ tokens) to Phi-3 Mini 128k model on laptop with Nvidia A2000 4GB RAM. At first about 3.3GB GPU RAM and 8GB CPU RAM is used by ollama, then the GPU ram usage slowly rises (3.4, 3.5GB etc.) and after about a minute it throws the error when GPU ram is exhaused probably (3.9GB is latest in task manager). The inference does not return any token (as answer) before crashing. Attaching server log. Using on Win11 + Ollama 0.1.42 + VS Code (1.90.0) + Continue plugin (v0.8.40).

The expected behavior would be not crashing and maybe rellocating the memory somehow so that GPU memory does not get exhausted. I want to disable GPU usage in ollama (to test for CPU inference only - I have 64GB RAM) but I am not able to find out how to turn the GPU off (even though I saw there is a command for it recently - am not able to find it again).

Actual error:

CUDA error: out of memory
  current device: 0, in function alloc at C:\a\ollama\ollama\llm\llama.cpp\ggml-cuda.cu:375
  cuMemSetAccess(pool_addr + pool_size, reserve_size, &access, 1)
GGML_ASSERT: C:\a\ollama\ollama\llm\llama.cpp\ggml-cuda.cu:100: !"CUDA error"

This is reported via Ollama and full logs are in the issue there: ollama/ollama#4985

Name and Version

See linked ollama issue.

What operating system are you seeing the problem on?

Windows

Relevant log output

See linked ollama issue.

[JohannesGaessler]

This is not necessarily a bug but rather an issue with how efficient resources are used, i.e. performance. And I can only speak for myself but I will definitely not debug the performance of downstream projects.

[zsogitbe]

There is definitely a bug in the GPU memory allocation code. We are experiencing several major issues:

• GPU memory is not fully allocated after loading the model, but it keeps allocating more when doing the first inference.
• The main problem with the above rises when there is no more GPU memory left in the main GPU for the context (kv cache?) to be fully allocated. This can happen at several occasions, for example, if the model does not fit into the GPU memory and only partially loaded, or if there are two models in GPU memory and the second model completely fills the main GPU memory. Etc. When the library tries to allocate the context it crashes because there is no more GPU memory left in the main GPU. The library crashes regularly while using GPUs.

The problem reported by kozuch is also because of this issue.

• One more issue we have found is that approximately 20% more GPU memory is used today than about 1 month ago with the same model (this has to do probably with the later GPU allocation of the context).

• One more issue is something strange with the split mode. When setting Row as split mode the number of off loaded layers still needs to be set. And while using the Layer split mode the library also crashes regularly and it uses more GPU memory compared to the Row split mode. Etc. I think that the GPU memory allocation should be completely reworked or validated in some way.

The right way of working should be that all GPU memory should be allocated when loading the model that it does not need to allocate more memory later (this would prevent the crashes).

[JohannesGaessler]

GPU memory is not fully allocated after loading the model, but it keeps allocating more when doing the first inference.

One more issue is something strange with the split mode. When setting Row as split mode the number of off loaded layers still needs to be set. And while using the Layer split mode the library also crashes regularly and it uses more GPU memory compared to the Row split mode. Etc.

That is just how those things are implemented. As of right now large batch matrix multiplication uses FP16 cuBLAS GEMM so you need allocate temporary buffers for the dequantized weight matrices. If you compile with LLAMA_CUDA_FORCE_MMQ you can use custom kernels that directly use the quantized data but there are still performance issues on Volta or newer.

And --split-mode row allocates the entire context on a single GPU so it's not surprising that you oom faster.

I think that the GPU memory allocation should be completely reworked or validated in some way.

I will happily review any PRs in that regard.

One more issue we have found is that approximately 20% more GPU memory is used today than about 1 month ago with the same model (this has to do probably with the later GPU allocation of the context).

That one could be an actual bug. If you want it fixed, provide instructions for reproduction and do a git bisect to identify the exact commit that introduced the increase in memory consumption.

[JohannesGaessler]

I forgot: enabling FlashAttention via -fa should also cut down a lot on temporary buffers needed for matrix multiplication. (Right now it cannot be the default because not all backends support it.)

[zsogitbe]

Thank you Johannes, I will try the FlashAttention option.

I am afraid that with the current GPU allocation code the library is not usable in production software unless you only use CPU (very slow). I know that the GPU part is a lot and complex work... hopefully will someone try to correct it. I would be willing to test it when it is ready.

For me this is a critical major issue, the library should have a basic correct working with multi-GPUs and CPUs. I see endless PRs with for me unimportant additions compared to this.

[JohannesGaessler]

For me this is a critical major issue, the library should have a basic correct working with multi-GPUs and CPUs. I see endless PRs with for me unimportant additions compared to this.

That is unfortunate but as it is there are only 2 devs working on CUDA (slaren and me). My work on the open llama.cpp repository is unpaid and I only really care about my own use cases. You could try a business inquiry if you really need specific changes.

[zsogitbe]

Thank you Johannes and of course also thank you @slaren for your contribution to the cuda part! I am not interested in the commercial options, I am here to support open source. I don't think that their business venture will be successful if the basics of the library are not good... the core should be robust and well working. The code is 5-10 times faster on the GPU when it does not crash, so for me the cuda part is important, your contribution is important! I am contributing to several open source projects, here I am mostly testing and trying to help with ideas and finding the right direction.

Since there is 20% more GPU memory allocation compared to one month ago, it is possible that someone tried to pre-allocate some memory for the context, but probably made a mistake somewhere. If you will have some time and interest to improve the cuda code in the library, then here are my comments:

• the main problem is the second allocation of GPU memory (probably the context - kv cache) because that causes major errors in several circumstances. I think that the code tries to allocate the GPU memory on the main GPU for the context and if there is no memory left there, then it just crashes The code should check if there is memory left on one of the available GPU's and if yes, then allocate the memory there or if there is nothing left, then allocate in system memory and use the CPU for that part, etc. maybe there are other optimized options.
  You can test the memory allocation amount by running an older version of the library and compare it to the current version, you will see the 20% more GPU memory use.
• The second allocation should be prevented and there should only be 1 allocation of GPU memory when the model is loaded. The context size should probably be fixed at loading time. The GPU may be used by several applications and models, so we cannot be sure that there will be memory left on the GPU for the second allocation.
• if possible the GPU memory use should be decreased back to the level where it was approximately one month ago. 20% more GPU memory use is too much.
• abort() in GGML_ASSERT(!"CUDA error"); should be replaced with throwing an exception in some way. A library which is just crashing out in this way is former century C style coding and should be prevented.

Could you please give us a summary of the compilation options with some explanation of what they do and their advantages and disadvantages, which could reduce the problems mentioned above? A bit more documentation would help and that is probably not a lot of work. You mentioned already:

• LLAMA_CUDA_FORCE_MMQ
• FlashAttention via -fa

What about GGML_CUDA_NO_VMM?

[JohannesGaessler]

I am contributing to several open source projects, here I am mostly testing and trying to help with ideas and finding the right direction.

The bottleneck is not testing or ideas, it is the amount of time that developers have. As I said, I will happily review any PRs that improve memory allocation since that will be significantly less work than having to implement it myself.

The code should check if there is memory left on one of the available GPU's and if yes, then allocate the memory there or if there is nothing left, then allocate in system memory and use the CPU for that part, etc. maybe there are other optimized options.

That is not going to work without major changes to the GPU code. It would be better to pre-allocate the memory.

The context size should probably be fixed at loading time.

It is. Any additional VRAM use comes from temporary buffers. With LLAMA_CUDA_FORCE_MMQ and -fa that should be 95% fixed.

if possible the GPU memory use should be decreased back to the level where it was approximately one month ago. 20% more GPU memory use is too much.

Since you said that you would help with testing, please do a git bisect and identify the bad commit.

abort() in GGML_ASSERT(!"CUDA error"); should be replaced with throwing an exception in some way. A library which is just crashing out in this way is former century C style coding and should be prevented.

llama.cpp (C++ user code) internally uses ggml (C library code).

What about GGML_CUDA_NO_VMM?

That's going to make it worse I think.

[ggerganov]

There is no "20% more GPU memory increase" - you are likely using a larger context (the default is now -c 0 so the full training context). In order for us to do anything, you need to provide specific repro steps using llama.cpp examples, otherwise we directly assume user error.

The memory estimation problem has been discussed in the past and it is not trivial to fix. Therefore, projects using llama.cpp have to know in advance the set of conditions that they will be running and constrain this set to be compatible with the underlying hardware. You cannot expect to be able to load arbitrary models in memory from different applications sharing the same GPU and everything to just run smoothly.

[zsogitbe]

if possible the GPU memory use should be decreased back to the level where it was approximately one month ago. 20% more GPU memory use is too much.

Since you said that you would help with testing, please do a git bisect and identify the bad commit.

I have done this already Johannes, here is the most probable reason: #6170