lightllm
lightllm copied to clipboard
Support disk radix cache
Must use with Radix Cache, will store all inputs into disk right after prefill (parallel), and when new request arrive, if disk cache len > gpu cache len && gpu cache len <= 0.5 * input_length, then pull from disk and use the disk cache.