Android Backend Support

[AmSmart]

As discussed in #695, Here's a PR that generates the shared library for Android. I have also added the new binaries to be copied in the CPU Backend .nuspec file. Please let me know if I am missing anything else.

I have tested the generated .so by loading manually into a .NET MAUI app which ran smoothly on my Android device. See video below

https://github.com/SciSharp/LLamaSharp/assets/26899350/d6afe4d9-21a4-455d-a74e-5a472249b2ea

[martindevans]

Looks good, thanks for putting this together.

Could you please manually trigger a run of the "Update Binaries" action on your repo as a test run. To do that:

1. Go to actions (top bar of your repo)
2. Select "Update Binaries" on the left
3. Click "Run Workflow" at the top of the run list (right side)
4. Change the branch to add-android-support
5. Click run workflow
6. Link the run here

[AmSmart]

Totally aligned. I was pondering that it deserved its own backend as well but wasn't sure. Will work on these over the next few days. Should share an update soon.

PS: Prefer to finish it up in this PR

[cmpktheo]

Could you upload the MAUI app's source with the android runtime libraries to test it?

[AmSmart]

Hi @cmpktheo Attaching a binary compliled for android-arm64-v8a

libllama.zip

[cmpktheo]

Thanks @AmSmart ! I have been trying to build llama.cpp on a linux VM and test a MAUI Android app with LlamaSharp using a Visual Studio x86_64 emulator.

I followed the android build instructions on llama.cpp page, but when I try to use the .so file, I get a null pointer calling the llama_model_default_params method (which basically returns a dummy structure) . It makes me think that there's either a flaw in my build process or some issue related to MAUI. This is my first attempt to build anything in MAUI so I might miss something. I've created a windows MAUI app that loads the GPT2 guff model and works perfectly.

I would appreciate if you could share the build process so I can apply it to x86_64 architecture or a ready build x86_64 runtime and ideally a MAUI-Android sample app.

[martindevans]

@cmpktheo did you ensure you're using the right version of llama.cpp? The C# source is only compatible with one exact version! See the table in the readme here for the correct versions.

[cmpktheo]

@martindevans the issue seems to be with my build process since the native method returns null pointer.

[AmSmart]

@cmpktheo You can check the compile.yml file from my PR. I've updated it to include other Android architectures.

[AmSmart]

@AsakusaRinne I've been a bit busy since our last conversation but I've finally gotten some time to address some of the concerns you listed earlier and made some other changes as well.

Things to note

• I have created a different Nuspec for the Android backend as discussed
• I noticed there was a .targets file but it wasn't been used by any of the generated packages (I double checked this by unzipping existing packages on Nuget). That said, I needed the targets file to Embed the library and link for Android so I removed all the dormant build directives in the file and have just the Android ones in it. I plan to open another PR for iOS and it would need the targets file as well. If you do not like these change, or need a bit more clarification. I'm happy to explain more
• Since Android doesn't support additional enhancements (CUDA, CUBLAS, AVX etc) at the moment, there is really no need to manually resolve the native library. I have added an if statement to exclude Android from the Manual Resolution in NativeApi.Load.cs
• Updated the compile.yml file to remove file extensions from directory names. I think it may confuse new contributors who may confuse the directories for files (I was confused for a moment).
• With regards to providing tests, I'm thinking of a world where we have a Demo/Example .NET Android Project instead. The UnitTests already provide coverage for the underlying code and a demo app should suffice. That said, we could still proceed to write tests against the demo app and have them run in the workflow. Let me know what you think, I'm happy to contribute this in another PR.

Let me know if there are anymore requested changes for this PR to be good to go.

[cmpktheo]

@cmpktheo You can check the compile.yml file from my PR. I've updated it to include other Android architectures.

@AmSmart , Your compiled .so file for the android-arm64-v8a architecture, is ~12 MB. When I run the Update Binaries workflow, the resulting .so file for the same architecture is ~17MB. I am asking because, when I use the compiled .so file for the x86_64 architecture I get a [libc] Fatal signal 11 error, which means that there's an issue with the build process or MAUI/.NET.

How did you get the 12MB version which is considerably smaller than my run of Update Binaries workflow?

[martindevans]

Since this PR has been stalled for a while I've taken the first step (generating the binaries) and added it in #861, hopefully we can start to make some progress on Android support :)

[martindevans]

Some of this work was pulled out into a separate PR (#861). The rest of the work will need to be updated to fix the conflicts in new PRs, so I'll close this one. It should still serve as a handy reference for future PRs.

[AmSmart]

Will pick up the rest of the effort in another PR as I find time.

[martindevans]

Hopefully I've got the most complex bit out of the way for you with the updated build script :)