mistral.rs
mistral.rs copied to clipboard
Update PyO3 to take dict
This increases compatibility with OpenAI and llama-cpp-python. I would appreciate any thoughts on this change.
Breaking
This breaks any code which uses the chat completion API as it removes the Message
and Role
classes.
Sample change
from mistralrs import Runner, Which, ChatCompletionRequest, Message, Role
runner = Runner(
which=Which.GGUF(
tok_model_id="mistralai/Mistral-7B-Instruct-v0.1",
quantized_model_id="TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
quantized_filename="mistral-7b-instruct-v0.1.Q4_K_M.gguf",
tokenizer_json=None,
repeat_last_n=64,
)
)
res = runner.send_chat_completion_request(
ChatCompletionRequest(
model="mistral",
- messages=[Message(Role.User, "Tell me a story about the Rust type system.")],
+ messages=[{"role":"user", "content":"Tell me a story about the Rust type system."}],
max_tokens=256,
presence_penalty=1.0,
top_p=0.1,
temperature=0.1,
)
)
print(res.choices[0].message.content)
print(res.usage)
Code Metrics Report
─────────────────────────────────────────────────────────────────────────────── Language Files Lines Blanks Comments Code Complexity ─────────────────────────────────────────────────────────────────────────────── Rust 70 23717 1560 511 21646 1313 ─────────────────────────────────────────────────────────────────────────────── Total 70 23717 1560 511 21646 1313 ─────────────────────────────────────────────────────────────────────────────── Estimated Cost to Develop 81,933 Estimated Schedule Effort 11.891481 months Estimated People Required 5.094738 ─────────────────────────────────────────────────────────────────────────────── Processed 784984 bytes, 0.785 megabytes (SI) ───────────────────────────────────────────────────────────────────────────────