lightllm icon indicating copy to clipboard operation
lightllm copied to clipboard

Support disk radix cache

Open jayfeather9 opened this issue 7 months ago • 0 comments

Must use with Radix Cache, will store all inputs into disk right after prefill (parallel), and when new request arrive, if disk cache len > gpu cache len && gpu cache len <= 0.5 * input_length, then pull from disk and use the disk cache.

jayfeather9 avatar Apr 18 '25 09:04 jayfeather9