mistral.rs icon indicating copy to clipboard operation
mistral.rs copied to clipboard

Update PyO3 to take dict

Open EricLBuehler opened this issue 9 months ago • 1 comments

This increases compatibility with OpenAI and llama-cpp-python. I would appreciate any thoughts on this change.

Breaking

This breaks any code which uses the chat completion API as it removes the Message and Role classes.

Sample change

from mistralrs import Runner, Which, ChatCompletionRequest, Message, Role

runner = Runner(
    which=Which.GGUF(
        tok_model_id="mistralai/Mistral-7B-Instruct-v0.1",
        quantized_model_id="TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
        quantized_filename="mistral-7b-instruct-v0.1.Q4_K_M.gguf",
        tokenizer_json=None,
        repeat_last_n=64,
    )
)

res = runner.send_chat_completion_request(
    ChatCompletionRequest(
        model="mistral",
-       messages=[Message(Role.User, "Tell me a story about the Rust type system.")],
+       messages=[{"role":"user", "content":"Tell me a story about the Rust type system."}],
        max_tokens=256,
        presence_penalty=1.0,
        top_p=0.1,
        temperature=0.1,
    )
)
print(res.choices[0].message.content)
print(res.usage)

EricLBuehler avatar Apr 30 '24 12:04 EricLBuehler

Code Metrics Report
  ───────────────────────────────────────────────────────────────────────────────
Language                 Files     Lines   Blanks  Comments     Code Complexity
───────────────────────────────────────────────────────────────────────────────
Rust                        70     23717     1560       511    21646       1313
───────────────────────────────────────────────────────────────────────────────
Total                       70     23717     1560       511    21646       1313
───────────────────────────────────────────────────────────────────────────────
Estimated Cost to Develop 81,933
Estimated Schedule Effort 11.891481 months
Estimated People Required 5.094738
───────────────────────────────────────────────────────────────────────────────
Processed 784984 bytes, 0.785 megabytes (SI)
───────────────────────────────────────────────────────────────────────────────
  

github-actions[bot] avatar Apr 30 '24 12:04 github-actions[bot]