Automatic Prefix Caching (#2792) might conflict with multi-LoRA (#1804)

[jacobthebanana]

#2762 Provides a great way to improve efficiency when multiple requests share the same prefix through KV-cache reuse. Nevertheless, the user probably does not want to share KV-cache across two different LoRA adapters since the values would most likely be different.

As the test cases in PR #3263 suggest, the code changes in #2762 might require a bit more work to distinguish between blocks from different LoRA adapters. Previously, #1804 avoided this conflict by including adapter_id in the tuple while generating hashes for prefixes. (source). The fix proposed in #3263 drew inspiration from this approach.

[jacobthebanana]

Resolved in #3263.

[justadogistaken]

@jacobthebanana
I wonder if it's optional to include adapter_id in the hashes for prefixes? I guess it should be enabled always, right? Because kvs should be different across different LoRA adapters.

[jacobthebanana]

@jacobthebanana I wonder if it's optional to include adapter_id in the hashes for prefixes? I guess it should be enabled always, right? Because kvs should be different across different LoRA adapters.

Exactly! I believe that when calculating hashes for the prefix, the code adds lora_int_id to the tuple to differentiate between adapters. If no adapter is currently active, the value of lora_int_id would be 0.

sequence.py#L237-L254