Eric Buehler

Sliding window models do not properly slice KV cache

2

**Describe the bug** This affects models which use sliding window attention, but only when the sequence length is great enough (seq_len > sliding_window) to need the sliding window. This will...

Default values in enum struct-variants

4

Hello everyone, Thank you for your great work here. Our project makes extensive use of struct enum variants. However, we have many variants which should have default values: some of...

Support quantized models

1

Refs #24.

Update PyO3 to take dict

1

This increases compatibility with OpenAI and llama-cpp-python. I would appreciate any thoughts on this change. # Breaking This breaks any code which uses the chat completion API as it removes...

Use indicatif progress bar instead of tqdm

1

This also updates the loading process to track loading of shards instead of tensors. This will enable loading in Jupyter without being rate limited and hanging.

Loading `tokenizer.model` with Rust API

10

Hello all, Thank you for your excellent work here. I am trying to load a `tokenizer.model` file in my Rust application. However, it seems that the `Tokenizer::from_file` function only support...

Implement dynamic LoRA swapping

1

Dynamic LoRA swapping, first raised in #259, enables the user to dynamically set active LoRA adapters. This can be configured per-request to enable users to add their own routing functionality....

`broadcast_as` error when processing multiple tokens at once in quantized example

9

Hello all, Thanks for your great work here. We are implementing speculative decoding at mistral.rs, and were in the final stages of testing when we discovered some incredibly strange behavior....