mlc-llm
mlc-llm copied to clipboard
mlc-ai
[Question] Support for Flutter
Hi @microcoder-py , are you referring to Google's Flutter SDK? https://github.com/flutter/flutter
Yes, that's the one. Now I know I could just write a platform native code for iOS/Android and bridge it to flutter, but doing it in dart would be much easier
Hi @microcoder-py , thanks for the feedback. If there is a C FFI for flutter, then it might be possible for us to provide flutter support. However, we don't have execution plans atm.
We also welcome the community to contribute. Here's the interface to MLC's C++ API: https://github.com/mlc-ai/mlc-llm/blob/main/cpp/llm_chat.cc
@Kathryn-cat there is a C FFI but we can write iOS and Android compatible code and bridge it to flutter. Since MLC already has Swift and Android support, maybe you could try only writing that bridge. I will try and do it myself but I am not a frontend dev
Hi, just checking if there are any updates on this @microcoder-py ? We’d be interested in helping if that is possible.
I'm also interested in Flutter support. I'm planning on giving it a spin myself but would be happy to collaborate if there's already an effort
@nundys terribly sorry I missed your mention, as of now I have veered away from the on device use case, but if there is anything I can do from my end to help happy to contribute
Hi @microcoder-py No problem, thanks for your response. @Decron, I haven't started yet but would be happy to join too.
@junrushao I am working on flutter support for mlc-llm. The first step is to create the dart bindings to https://github.com/mlc-ai/mlc-llm/blob/main/cpp/llm_chat.cc Unfortunately the ffigen tool does not work well with the header: https://github.com/mlc-ai/mlc-llm/blob/main/cpp/llm_chat.h Could you help point to the relevant functions in the cc file which will need bindings? It looks like I will need to hand roll them. Any help appreciated.
@anushreedhople I am very excited to try out your work! How's the progress?
I would also love to try your progress! Locally running LLMs with flutter would be a dream come true!
After reading a bit more, I think easiest way would to to interface the TVM C API and work through https://dart.dev/guides/libraries/c-interop.
The rust module might serve as a reference https://github.com/mlc-ai/mlc-llm/pull/1213. Although atm we do not have planned effort on this direction
