llama.cpp
llama.cpp copied to clipboard
ggml-org
Add support for external call from iOS, Android, macOS and Flutter
Hi,
Im making a Flutter app that call the main function from llama.cpp, but im getting some crash. Can you add a target for CMake to i build shared libraries that have a public function that send the data to socket?
My Flutter app is working here: https://github.com/paulocoutinhox/llama-flutter
My fork with required patches is here (but i patch only to my case): https://github.com/paulocoutinhox/llama.cpp/pull/1/files
If you create a CMake target that i don't need patch and have C/C++ interfaces that let us call the function to execute the prompt, i can use on my Flutter project and use mine more robust project called XPLPC (https://github.com/xplpc/xplpc).
Thanks for the help.
You can already compile a library to call from other languages with just a C ABI. main.cpp has a lot of stuff in there that you don't really want if you're integrating it into another program, like the terminal handling, reading from stdin, etc.
If you configure CMake with -DBUILD_SHARED_LIBS=ON then you get a .so/.dll/.dylib. This can be linked into your program. I know I have done this successfully from C#, for example. Another example is the Python wrapper: llama-cpp-python.
The only difference is that you have to use the lower level functions to generate text: tokenize, eval, sample, etc. And manage the context so it doesn't fill.
Hi,
Yes, i see that projects. The problem in that way of use is that i need create or reimplement most things in Dart.
I want use it like the executable from terminal, where we can pass some parameters and it works, like the executable generated.
Will be nice if we have a way to call like this, because all the implementation will stay in C++ side.
In the fork at llama-jni, we rewrite main.cpp to provide the JNI for Android applications, basically achieving to get every text output of the compiled main.cpp.
On this basis, we also implemented llama-pinyinIME, which is an IME with LLM capability.
Hope it might help.
yes, need iOS and Android suppor too!
Soon I will bring the core module to a separate repository, for now you can look here, maybe you will find something useful for iOS.
Probably easiest right now is the server example. Currently is running on TCP sockets, but maybe Unix sockets or named pipes is possible.
Note that server currently does not support authentication, so other apps on the same device may be able to access it.
