[Feature]: Estimate max-model-len when the KV cache memory is not enough

[heheda12345]

🚀 The feature, motivation and pitch

When the KV cache is not enough for holding one request, vLLM v1 will raise an error like this

ERROR 04-05 01:12:55 [core.py:390] ValueError: To serve at least one request with the models's max seq len (1048576), (24.00 GiB KV cache is needed, which is larger than the available KV cache memory (9.97 GiB). Try increasing gpu_memory_utilization or decreasing max_model_len when initializing the engine.

It would be more convenient if we can provide an estimated max_model_len to the users in this error log.

The estimation is more complex than max_model_len = block_size * num_gpu_blocks after the introduction of different types of KV cache like sliding window, and help wanted on implementing with binary search of max_model_len based on the KVCacheSpec.max_memory_usage_bytes.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[lengrongfu]

I have implemented a basic code draft, is this what you want? @heheda12345 , if yes, can assign me, to implement it.
define estimate_max_model_len method, than in check_enough_kv_cache_memory call:

if needed_memory > available_memory:
        # Estimate the maximum model length that can fit in the available memory
        estimated_max_len = estimate_max_model_len(vllm_config, kv_cache_spec, available_memory)
        estimated_msg = ""
        if estimated_max_len > 0:
            estimated_msg = f" Based on the available memory, the estimated maximum model length is {estimated_max_len}."

• def estimate_max_model_len

def estimate_max_model_len(vllm_config: VllmConfig,
                        kv_cache_spec: dict[str, KVCacheSpec],
                        available_memory: int) -> int:
   
    # Define a function to check if a given model length fits in memory
    def fits_in_memory(model_len: int) -> bool:
        # Create a deep copy of the config to avoid modifying the original
        temp_config = copy.deepcopy(vllm_config)
        # Modify the max_model_len for this calculation
        temp_config.model_config.max_model_len = model_len
        # Calculate memory needed for the given model length
        memory_needed = 0
        for layer_spec in kv_cache_spec.values():
            memory_needed += layer_spec.max_memory_usage_bytes(temp_config)
        return memory_needed <= available_memory

    # Binary search for the maximum model length
    # Start with a reasonable range - from 1 to current max_model_len
    current_max = vllm_config.model_config.max_model_len
    left, right = 1, current_max

    # If even the smallest model length doesn't fit, return 0
    if not fits_in_memory(left):
        return 0

    # If the maximum model length fits, no need for binary search
    if fits_in_memory(current_max):
        return current_max

    # Binary search for the maximum model length that fits
    result = 1
    while left <= right:
        mid = (left + right) // 2
        if fits_in_memory(mid):
            result = mid
            left = mid + 1
        else:
            right = mid - 1

    return result

[blumfontein]

Hi, I tried to launch new Llama4 Scout on 8 H100 and got ther error

ValueError: To serve at least one request with the models's max seq len (10485760), (240.00 GiB KV cache is needed, which is larger than the available KV cache memory (98.16 GiB). Try increasing gpu_memory_utilizationor decreasingmax_model_len when initializing the engine.

My thoughts were 640GB should be ok to host 109B model FP16.
Ok, then I tried to launch it on 8 H200, but get exact same error.
Where did 1T VRAM spend for?

8 h200 must be more than enough to host it without decreasing max_model_len. What do I do wrong? My command is

docker run -d --restart unless-stopped --runtime nvidia --gpus all --name llama4-scout -v ~/.cache/huggingface:/root/.cache/huggingface --env "HUGGING_FACE_HUB_TOKEN=123" -p 8081:8000 --ipc=host vllm/vllm-openai:latest --model meta-llama/Llama-4-Scout-17B-16E-Instruct --tensor-parallel-size 8 --gpu-memory-utilization 0.98

[heheda12345]

I have implemented a basic code draft, is this what you want? @heheda12345 , if yes, can assign me, to implement it.
define estimate_max_model_len method, than in check_enough_kv_cache_memory call:

@lengrongfu Thanks for your quick response. Your implementation is almost correct, but the code needs some cleanup and unit tests. Can you create a PR for it?

[heheda12345]

@blumfontein please use the example command for 8 h100 in https://blog.vllm.ai/2025/04/05/llama4.html . Llama 4's context length is quite large and you need to set the max-model-len manually to reduce the needed kv cache.

[lengrongfu]

I have implemented a basic code draft, is this what you want? @heheda12345 , if yes, can assign me, to implement it.
define estimate_max_model_len method, than in check_enough_kv_cache_memory call:

@lengrongfu Thanks for your quick response. Your implementation is almost correct, but the code needs some cleanup and unit tests. Can you create a PR for it?

OK, I will submit a pr in when it is more perfect.
Please assign me.