Eric Buehler
Eric Buehler
It seems like the CI tests are failing because this integration depends on the `mistralrs` library. Would it be best if I update the CI to install and build `mistralrs`?
@nerdai, I just released `mistralrs` on pypi. However, it requires a Rust toolchain to build. Can we update the CI to install the Rust toolchain?
@nerdai, yes, it can. It depends on `openssl` though.
@nerdai, thanks for clarifying! I have added it as a dependency now.
Hi @nerdai! I have updated this PR to use our latest PyPi release. Additionally, I made sure the tests pass by running the following commands: ``` make format make lint...
@nerdai, that should be fixed now! Not sure why the tests I ran locally didn't catch that though.
@nerdai thank you!
@lucasavila00, thanks for your work here. I think we definitely need to pursue the throughput decrease as the batch increase. I like the idea of async being incorporated, did this...
> On bs=8 it improved from 5ms to 2ms. Great! Perhaps we could profile it and see when the performance gains go away, and use this then. > Clippy raised...
@lucasavila00 > cargo test fails because it can't download the mistral tokenizer from HF. I set the `HF_TOKEN` secret during CI: > https://github.com/EricLBuehler/mistral.rs/blob/9ec90a40949efe9c68828c048da136874d1ba2f1/.github/workflows/ci.yml#L49