[resouce manager] Fix free memory fraction calculation

Respect fraction specified, or else we will disregard the memory
taken for storing the model and get out-of-memory (OOM) for
allocating too much blocks.

Signed-off-by: eopXD <[email protected]>

Author: eopXD

tensorrt_llm/_torch/pyexecutor/resource_manager.py

@@ -814,7 +814,14 @@ def calculate_max_num_blocks_from_cpp(
         logger.debug(f"window_size_to_layers: {window_size_to_layers}")
 
         free_mem, total_mem = torch.cuda.mem_get_info()
-        primary_pool_memory_bytes = free_mem
+        free_mem_fraction = (kv_cache_config.free_gpu_memory_fraction
+                             if kv_cache_config.free_gpu_memory_fraction
+                             is not None else 0.9)
+        assert free_mem_fraction < 1.0, (
+            f"Invalid freeMemFraction: {free_mem_fraction} must be < 1.0")
+        logger.debug(f"free_mem_fraction: {free_mem_fraction}")
+
+        primary_pool_memory_bytes = int(free_mem * free_mem_fraction)
         secondary_pool_memory_bytes = 0
         logger.debug(
             f"primary_pool_memory_bytes is set to {primary_pool_memory_bytes/1024**3}GB, \n"