llama.cpp
llama.cpp copied to clipboard
ggml-org
Go bindings
Hey :wave: , awesome project!
I'd like to help here, I've did the bindings for go to be used as a library. there are of course some adaptations that I had to run into to make it possible. I'm wondering, there are any plans for golang bindings? Generally speaking there seems to be genuine interest into running this as API too https://github.com/antimatter15/alpaca.cpp/issues/86 , which I worked on here: https://github.com/go-skynet/llama-cli .
I'm happy to contribute my work which currently is in https://github.com/go-skynet/llama, I'm playing with this sparely - I've been using this mainly with alpaca, and could manage to run also 13b and 30b models. I'll update it later to the latest changes so it is on pair with master.
Alright, a polished version using the new lib interfaces is here: https://github.com/go-skynet/go-llama.cpp/tree/master @ggerganov @gjmulder I'd be more than happy to have it here instead - would make it sense to wrap it up in a PR?
@mudler Looks great! If you wish to add it to this project, please see how we organized the Go bindings in the whisper.cpp repo and provide basic CI scripts together with the change. I am also happy to link to your repo from the README if you this would be easier for you.
I'm happy either way to be honest, I think mostly depends if you plan to have the language bindings sitting nearby the llama.cpp codebase. I was planning in any case to add documentation to the bindings.
I think it would best for the bindings to sit next to the project for maintainability, however maybe its worth meanwhile I work on a PR we can add it to the README, what do you think?
I'm checking whisper.cpp and I see it does tests also with the model, which I think we can't do here with llama.cpp as most of weights/models I can find are bind to the LLaMA license, which is not permissive - it prohibits redistribution and modifications (so any variant would be against their license, potentially we could violate DCMA just by referencing links to the model).
What do you suggest here? A simple test that compiles the binding and exercise the lib minimally would be ok?
I was looking for Go bindings today, but in the go-skynet version, the llama_predict wrapper function was too high level for what I wanted. If it helps at all for reference, I've also done a basic Go app using the llama.h functions directly - https://git.ivysaur.me/code.ivysaur.me/llamacpphtmld although there's no useful binding abstraction at all.
I was looking for Go bindings today, but in the go-skynet version, the
llama_predictwrapper function was too high level for what I wanted. If it helps at all for reference, I've also done a basic Go app using the llama.h functions directly - https://git.ivysaur.me/code.ivysaur.me/llamacpphtmld although there's no useful binding abstraction at all.
good food for thoughts, I was thinking too to expose a lower level API for granular control in go as well aside a higher level one, maybe we can join forces and do it together, WDYT?
Very high level, but I thought I would leave this here anyway: https://github.com/matthewrennie/go-llama.cpp
