mistral.rs icon indicating copy to clipboard operation
mistral.rs copied to clipboard

Add Rust convenience macros for loading

Open EricLBuehler opened this issue 1 year ago • 1 comments

use mistralrs::{device, load_normal_model};
fn main() -> anyhow::Result<()> {
    let dev = Device::cuda_if_available(0)?;
    let runner = load_normal_model!(
        id = "mistralai/Mistral-7B-Instruct-v0.1".to_string(),
        kind = Mistral,
        device = dev,
        use_flash_attn = false
    );
    Ok(())
}

EricLBuehler avatar Jun 20 '24 19:06 EricLBuehler

Code Metrics Report
  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 Dockerfile              1           34           25            0            9
 Happy                   1          442          369            0           73
 JSON                    9           21           21            0            0
 Python                 32         1256         1075           37          144
 TOML                   16          444          403            1           40
-------------------------------------------------------------------------------
 Jupyter Notebooks       1            0            0            0            0
 |- Markdown             1           60           30           22            8
 |- Python               1           96           87            1            8
 (Total)                            156          117           23           16
-------------------------------------------------------------------------------
 Markdown               17         1297            0          965          332
 |- BASH                 5          100           97            0            3
 |- Python               6          122          110            0           12
 |- Rust                 3          151          135            6           10
 (Total)                           1670          342          971          357
-------------------------------------------------------------------------------
 Rust                  119        36038        32599          623         2816
 |- Markdown            59          658           13          608           37
 (Total)                          36696        32612         1231         2853
===============================================================================
 Total                 197        39532        34492         1626         3414
===============================================================================
  

github-actions[bot] avatar Jun 20 '24 19:06 github-actions[bot]