GPU Memory Leak

[zsogitbe]

After disposing everything the GPU memory is still not freed. When running the same code for the second time it crashes with a message that it cannot allocate enough memory. There are some things which are not freed from GPU memory.

You can test the issue by using ggml_backend_cuda_get_device_memory(0, out freemem, out totalmem); before and after using a llama model. freemem will show how much memory is still allocated.

See this also: SciSharp/LLamaSharp#575

[zsogitbe]

I have noticed that cublasCreate is called, but there is no cublasDestroy. Could this be the problem?

[zsogitbe]

See also: #5576

[zsogitbe]

Added a new PR with corrected Cublas release GPU memory code: #5898

[ggerganov]

I don't understand how a process that has died could still occupy GPU memory. This looks like a non-llama.cpp related issue

[zsogitbe]

You are obviously a genius Georgi, so I am sure that you will figure it out. Please run the main example code in llama.cpp, free all memory and then stop at the end with debugging and check your GPU memory status. You will see that GPU memory is not freed.
This is a huge problem for production applications because GPU memory is very expensive and precious and you cannot just keep filling it with not used things.
We need to load the library and then allow for several inferences, but if each inference leaves hundreds of MBs GPU memory behind then you will soon not be able to do inference on the GPU...

In this example nearly 1GB is stayed on the GPU after running the inference and it is impossible to free it without exiting the app (main process end).

[ggerganov]

I misunderstood that the GPU memory stayed occupied after the process ends.

Currently, the CUDA backend cannot be cleaned-up completely because we have some global objects which are never destroyed. The problem is acknowledged and related to #3960. Keep track of that issue for further updates on this

[zsogitbe]

Thank you for acknowledging it. This is really a huge issue if you want llama.cpp being used in professional production software and not only in playing with llms. I have added a PR for releasing Cublas, trying to help (this freed 15% of the still occupied GPU memory), but I was not able to find the other 85%. I hope that you will find some time for this and give this a high priority. This problem should be solved before any further improvements to the library.

[zsogitbe]

Update: LLava is even a bigger problem because when using it the not freed GPU memory doubles (1.8 GB GPU memory is not freed in the above example)!

[slaren]

We need to load the library and then allow for several inferences, but if each inference leaves hundreds of MBs GPU memory behind then you will soon not be able to do inference on the GPU

To clarify, this is not what happens. Some resources are never freed but they are not leaked either, they are reused for future inferences. Memory usage does not increase indefinitely.

[zsogitbe]

Thank you for the clarification slaren, I can confirm that this is the case. But, it seems that Llava allocates a separate second pool of GPU memory because the not freed GPU memory doubles when using Llava. This was the reason for why I first thought that it keeps allocating extra GPU memory.

[zsogitbe]

slaren, any news on this? Do you think that you will have some time to fix the CUDA backend?

[ggerganov]

Currently, the CUDA backend cannot be cleaned-up completely because we have some global objects which are never destroyed. The problem is acknowledged and related to #3960. Keep track of that issue for further updates on this