Add Llama.cpp integration

[davidmigloz]

• LlamaCpp
• https://github.com/netdur/llama_cpp_dart
• Tool calling:
  • https://github.com/ggerganov/llama.cpp/issues/5588
  • https://www.youtube.com/watch?v=rsDlu-9UP00

[danemadsen]

I was helping develop llama_cpp_dart for a while but i was having too many issues with it so i ended up creating my own which works alot better.

https://github.com/Mobile-Artificial-Intelligence/maid_llm

If you want to integrate that into langchain you should start there

[davidmigloz]

Ah I didn't know you dropped llama_cpp_dart, I'll check maid_llm out. Thanks for bringing this up @danemadsen!

If we integrate llama.cpp into LangChain.dart, I guess you could simplify maid implementation a bit, as all the providers will use the same standard interface LangChain.dart provides.

After I'm done with the v0.5.0 release, I'll dig into it.

[danemadsen]

Yeah, I did actually have it using langchains ChatMessage at one point but I changed it due to a few little things I wanted from my own ChatNode types.

llama_cpp_dart was a decent implementation, its probably the fastest implementation I've used thusfar but I was running into a few different sampling issues I simply couldn't fix in that implementation so I just ended up creating my own from scratch.

I've got vulkan acceleration working in my own now so yeah its abit better

[davidmigloz]

Good to know. If you miss any functionality in the current abstractions let me know, and we can add it.

v0.5.0 breaks langchain into langchain_core, langchain and langchain_community, so you could depend just on langchain_core if you only need to use the base abstractions without depending on the entire langchain package.

[danemadsen]

Good to know. If you miss any functionality in the current abstractions let me know, and we can add it.

v0.5.0 breaks langchain into langchain_core, langchain and langchain_community, so you could depend just on langchain_core if you only need to use the base abstractions without depending on the entire langchain package.

Nah its probably nothing that really needs to be added into langchain. Its just that I'm using a tree structure for my chat messages so I can have branching conversations. You can add that in if you want it but your chatmessages are fine as is for most cases.

[raymon-io]

Is there any update on this? If there is some documentation for maid_llm , I could try to help integrate it into LangChain.

[davidmigloz]

@raymon-io I haven't started working on it yet, so it'd be great if you pick it up 🙂

We may have to create a langchain_community_flutter package to add integrations that depend on the Flutter SDK (as the current langchain_community package is a pure Dart one).

Happy to discuss it if you are interested!