[Feature]: Chunked prefill + lora

[rkooo567]

🚀 The feature, motivation and pitch

Currently lora doesn't work with chunked prefill because some of lora index logic doesn't cover the case where sampling is not required. This also means lora is not working with sampling_params do_sample=True.

We need to add test cases for these. WIP https://github.com/vllm-project/vllm/pull/4994

Alternatives

No response

Additional context

No response

[rohithkrn]

@rkooo567 can you share an example to reproduce this issue?

[rkooo567]

I think you can simply create a test case by adding chunked prefill to any lora correctness test!

[rkooo567]

https://github.com/vllm-project/vllm/tree/main/tests/lora

[rohithkrn]

@rkooo567 actually, when I run tests/lora/test_llama.py it passed. However, when I run examples/multilora_inference.py with chunked prefill the results are not matching results without chunked prefill. So, want to make sure we are talking about the same issue, I am trying to look into this on my side as well.

[rohithkrn]

@rkooo567 also are you seeing garbage output or an error?

[sfc-gh-zhwang]

you mean This also means lora is not working with sampling_params do_sample=False.? @rkooo567

[rkooo567]

@rkooo567 actually, when I run tests/lora/test_llama.py it passed. However, when I run examples/multilora_inference.py with chunked prefill the results are not matching results without chunked prefill. So, want to make sure we are talking about the same issue, I am trying to look into this on my side as well.

Hi, I just this. I think the loral + chunked prefill now is basically broken because lora assumes some index mapping that only works with default scheduling policy. I think the side effect could be wrong output or crash

[rkooo567]

you mean This also means lora is not working with sampling_params do_sample=False.? @rkooo567

Yes! that's right

[Nero10578]

Will chunked prefill ever work with LORA?

[sfc-gh-zhwang]

see https://github.com/vllm-project/vllm/pull/9057 @Nero10578

[mces89]

@sfc-gh-zhwang is there any plan when this feature can be merged?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions[bot]]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!