llm

llm copied to clipboard

Not sure if we should consider this out of scope, but bloomz.cpp is a fork of llama.cpp that's capable of inference with the BLOOM family of models. The changes don't look very large, so there's room for code sharing here: https://github.com/NouamaneTazi/bloomz.cpp/commits/main?before=ade8a9d82fa1dc440c26f09a9e02cc94d7294251+35&branch=main&qualified_name=refs%2Fheads%2Fmain

Even if we don't support it directly, it may be worth publishing a safe-ish version of ggml-rs to crates.io so that a library like llama-rs could be built for BLOOM.

I'd say this is in-scope for the project, but I don't have enough time to tackle this unfortunately :sweat_smile:

PRs welcome for anyone who wants to take on the task!

Ill try my hand at this one

Current state of affairs: this was implemented in #85, but that went out of sync after recent changes. It was then updated and trait-ified in #141, which is on track to be merged soon 🚀

Done! Sort of... I'll open a new issue.