mistral.rs
mistral.rs copied to clipboard
EricLBuehler
Blazingly fast LLM inference.
## Describe the bug I ran the following, and it downloaded both uqff as well as full weights from Meta. I tried to skip -m, but it seems -m is...
How is it possible to load multiples images for Phi-3.5-vision-Instruct ? And referencing them as Image ? Maybe it is supported but now example to show how.
## Describe the bug Mistral-server in interactive mode cannot read the image file path with space. I tried with `/file path`, `"/file path"`, `/file\ path`, and none of them worked....
## Changes - The core change here is ```diff -pub type MessageContent = Either; +pub type MessageContent = Either; ``` - Also, change `BasicProcessor` to use `template_action() -> MessagesAction::Keep` (instead...
## Describe the bug - To follow OpenAI spec (and what other open source models, e.g. llama3.1), expect for Assistant messages with tool calls, messages should look like this ```json...
# Description Loading a simple quantized GGUF model with CUDA fails with: **Error: DriverError(CUDA_ERROR_OUT_OF_MEMORY, "out of memory")** [log.txt](https://github.com/user-attachments/files/17238700/log.txt) # Environment - Linux Mint 21.3 Virginia - GeForce RTX 2060 SUPER...
