mistral.rs
mistral.rs copied to clipboard
Store and load prefix cache on disk
This PR enables storing and then restoring the model-specific prefix cache on disk. The intended use case, paired with #350, is to accelerate few-shot learning use cases by allowing a bank of prefixes which may be saved and loaded at runtime (optionally at user's discretion).
Code Metrics Report
=============================================================================== Language Files Lines Code Comments Blanks =============================================================================== Dockerfile 1 34 25 0 9 Happy 1 442 369 0 73 JSON 9 21 21 0 0 Python 21 741 622 21 98 TOML 15 393 355 1 37 ------------------------------------------------------------------------------- Jupyter Notebooks 1 0 0 0 0 |- Markdown 1 60 30 22 8 |- Python 1 96 87 1 8 (Total) 156 117 23 16 ------------------------------------------------------------------------------- Markdown 16 1054 0 781 273 |- BASH 6 203 190 0 13 |- Python 6 121 110 0 11 |- Rust 3 185 172 9 4 (Total) 1563 472 790 301 ------------------------------------------------------------------------------- Rust 86 28468 26049 381 2038 |- Markdown 42 440 0 428 12 (Total) 28908 26049 809 2050 =============================================================================== Total 151 31153 27441 1184 2528 ===============================================================================