flashinfer icon indicating copy to clipboard operation
flashinfer copied to clipboard

Published 20 hours ago • verified publisher icon flashinfer-ai

Feature/non contiguous kv cache

Open LinHeLurking opened this issue 4 months ago • 2 comments

This PR solves #506

Custom strides to support non-contiguous kv cache. Tests in test_batch_prefill_kernels.py and test_batch_decode_kernels.py are modified to test input kv_data on both contiguous and non-contiguous tensor.

LinHeLurking avatar Sep 29 '24 03:09 LinHeLurking