mistral.rs icon indicating copy to clipboard operation
mistral.rs copied to clipboard

Store and load prefix cache on disk

Open EricLBuehler opened this issue 1 year ago • 1 comments

This PR enables storing and then restoring the model-specific prefix cache on disk. The intended use case, paired with #350, is to accelerate few-shot learning use cases by allowing a bank of prefixes which may be saved and loaded at runtime (optionally at user's discretion).

EricLBuehler avatar May 30 '24 17:05 EricLBuehler

Code Metrics Report
  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 Dockerfile              1           34           25            0            9
 Happy                   1          442          369            0           73
 JSON                    9           21           21            0            0
 Python                 21          741          622           21           98
 TOML                   15          393          355            1           37
-------------------------------------------------------------------------------
 Jupyter Notebooks       1            0            0            0            0
 |- Markdown             1           60           30           22            8
 |- Python               1           96           87            1            8
 (Total)                            156          117           23           16
-------------------------------------------------------------------------------
 Markdown               16         1054            0          781          273
 |- BASH                 6          203          190            0           13
 |- Python               6          121          110            0           11
 |- Rust                 3          185          172            9            4
 (Total)                           1563          472          790          301
-------------------------------------------------------------------------------
 Rust                   86        28468        26049          381         2038
 |- Markdown            42          440            0          428           12
 (Total)                          28908        26049          809         2050
===============================================================================
 Total                 151        31153        27441         1184         2528
===============================================================================
  

github-actions[bot] avatar May 30 '24 17:05 github-actions[bot]