LLamaSharp icon indicating copy to clipboard operation
LLamaSharp copied to clipboard

Published 20 hours ago • verified publisher icon SciSharp

Feature/add android support

Open AmSmart opened this issue 8 months ago • 20 comments

Description of Changes

  • Updated compile Android to re-enable build and fix pipeline errors
  • Added Android native libs to the CPU backend .nuspec file
  • Updated .targets file to include new Android additions
  • Added an extra condition to skip custom DLL resolution on Android because Android auto resolves libs based on generated apk
  • Add Maui test Project (only Android enabled at the moment, will enable iOS in future PR)

Please let me know if any updates are required.

AmSmart avatar Mar 19 '25 06:03 AmSmart

cc: @martindevans

AmSmart avatar Mar 19 '25 06:03 AmSmart

Thanks for the hard work @AmSmart! I'll try to find some time over the weekend to give this a proper in depth review.

Looking at the CI failures, it looks like some new workloads might need to be installed to compile the project. That'll require some changes in this file I would guess.

martindevans avatar Mar 20 '25 01:03 martindevans

Is it CPU inference only or can we have GPU acceleration as well?

amirvenus avatar Mar 20 '25 23:03 amirvenus

It's only CPU in this PR.

As far as I know, Mobile GPU inference is still quite slow at the moment. Saw it was even slower than CPU inference in some cases. Besides, it takes a bit of effort building CLBlast from source and trying to get that to work with the Android builds so I didn't consider it.

AmSmart avatar Mar 21 '25 06:03 AmSmart

I've given this a deeper look, overall it looks good!

Left to resolve:

  • [ ] Fix those CI errors (workloads must be installed: maui-android)
  • [ ] Change nuget package to split out Android binaries from default ones
  • [ ] Merge conflict
    • If you're splitting up the nuget this should be easy, just revert all changes to the backend.CPU.nuspec
  • [x] Test run (actually just found this run, so it's done)

martindevans avatar Mar 23 '25 00:03 martindevans

Great, thanks for the review. Will follow up with the requested changes later today.

AmSmart avatar Mar 23 '25 01:03 AmSmart

Hello @martindevans, I've made updates to the PR. Looks like we're good to go. Not sure how you trigger the Unit Test pipelines in this repo, though, to get a finite confirmation.

AmSmart avatar Apr 19 '25 12:04 AmSmart

Thanks @AmSmart, I'll be doing an update to the llama.cpp version soon (which will involve generating new binaries). I'll be basing that PR off this, so it should all be merged in then.

martindevans avatar Apr 19 '25 16:04 martindevans

Great. Glad to hear it

AmSmart avatar Apr 20 '25 00:04 AmSmart

Will this include VLM inference as well for multimodal models such as Qwen2.5-VL 0.5B or Gemma3 3b?

Thanks

amirvenus avatar Apr 20 '25 02:04 amirvenus

@AmSmart I've merged this work into a WIP branch and done a test run. Unfortunately one of the Android targets failed, could you take a look at it (here)? if you have any fixes to PR please target them at that branch.

martindevans avatar Apr 21 '25 14:04 martindevans

I'll take a look at it

AmSmart avatar Apr 21 '25 16:04 AmSmart

It looks like clip.cpp is incompatible with 32 bit architectures, so we might have to disable the x86 target. That doesn't seem like a huge loss, I can't imagine there are many x86 Android devices out there?

martindevans avatar Apr 21 '25 18:04 martindevans

Ahh, I see. Not surprising at all. Majority of Android devices are arm64, so you're right about it mostly being an acceptable/negligible loss.

PS: I even had to disable building some of the binary examples because it didn't support x86. It's mostly a relic of the past at this point.

AmSmart avatar Apr 21 '25 19:04 AmSmart

Ok I disabled the x86 target and started a new test run: https://github.com/SciSharp/LLamaSharp/actions/runs/14580598934

martindevans avatar Apr 21 '25 20:04 martindevans

I've just been working on the new binary update (which includes this work). But when I tried to build LLama.Mobile I received this error: Specified AndroidManifest file does not exist: <redacted>\LLamaSharp\Llama.Mobile\AndroidManifest.xml.

Any ideas?

martindevans avatar Apr 28 '25 23:04 martindevans

Are you building directly on my branch or did you merge to a staging branch? I'd like to reproduce the issue

AmSmart avatar Apr 29 '25 12:04 AmSmart

I'm currently working with this branch: https://github.com/martindevans/LLamaSharp/tree/update_apr_2025

martindevans avatar Apr 29 '25 12:04 martindevans

Hi @martindevans

Took a look at this and seem to have figured it out. Inspecting the platform folder ../Llama.Mobile/Platforms/Android, I noticed that some things were different from my local vs your online branch. Apparently, the .gitignore file excludes all xml files (training model resources section) from the repo and as a result, the required xml files never got in to the PR.

Is it okay to remove the *.xml line from the gitignore file? Or can you share a more specific ignore rule so I can update my branch of the PR with the missing config files.

AmSmart avatar Apr 30 '25 12:04 AmSmart

I think it should be fine to remove that line, just keep an eye on what's being added after you remove it.

When you make this change, could you please PR it to my branch along with the extra files? Thanks.

martindevans avatar Apr 30 '25 13:04 martindevans

PR all done now @martindevans

AmSmart avatar May 01 '25 03:05 AmSmart

Thanks

martindevans avatar May 01 '25 14:05 martindevans

I've just opened https://github.com/SciSharp/LLamaSharp/pull/1179 which includes this work, so I'll close this PR now. Thanks for all the hard working getting Android ready!

martindevans avatar May 01 '25 14:05 martindevans

Amazing, it's my pleasure 🙌🏽

AmSmart avatar May 01 '25 14:05 AmSmart

I've just opened #1179 which includes this work, so I'll close this PR now. Thanks for all the hard working getting Android ready!

Was android merged at the end or was it discarded?

amirvenus avatar May 11 '25 20:05 amirvenus

#1179 was merged just a couple of hours ago, so Android support is now in the master branch but not yet on nuget.

martindevans avatar May 11 '25 20:05 martindevans

#1179 was merged just a couple of hours ago, so Android support is now in the master branch but not yet on nuget.

Excellent! I can’t wait to try it out!

Do you know that VLMs such as InternVL are also supported? Thanks

amirvenus avatar May 11 '25 20:05 amirvenus