tvm icon indicating copy to clipboard operation
tvm copied to clipboard

[Runtime] Fix PagedKVCache for PopN and enhance tests

Open MasterJH5574 opened this issue 1 year ago • 0 comments

This PR fixes a bug in the PagedKVCache which may happen when the sequence removal order is not consistent with the reverse order of sequence add/fork order. With this fix, the PagedKVCache now supports removing sequences in any order without breaking.

This PR also adds an empty function to PagedKVCache to check if the KV cache is empty. Right now this function is only used for test purpose, where we check if everything in the KV cache is freed after removing all sequences.

MasterJH5574 avatar May 29 '24 21:05 MasterJH5574