Add GPU support to LLM C API

[huazhusun]

Here I use bazel build --config=android_arm64 //mediapipe/tasks/java/com/google/mediapipe/tasks/genai:libllm_inference_engine_jni.so to get libllm_inference_engine_jni.so , I use it in my application, it's crashed. 2024-04-09 18:25:38.304 3468-3678 native com.example.scigemma A F0000 00:00:1712658338.304137 3678 llm_inference_engine_cpu.cc:263] Failed to create session: INTERNAL: ; RET_CHECK failure (mediapipe/tasks/cc/genai/inference/c/llm_inference_engine_cpu.cc:180) (backend)==("cpu")

However, I using https://mvnrepository.com/artifact/com.google.mediapipe/tasks-genai , it's success.

It's same error when using the file tasks_genai.aar built with bazel build -c opt --config=android_arm64 --linkopt="-s" //mediapipe/tasks/java/com/google/mediapipe/tasks/genai:tasks_genai

by the way, the file of " tasks_genai.aar" size I built is 9,823,664, the file size in maven is 5,553,876

Are there some difference ? My compiling steps are: git clone https://github.com/google/mediapipe.git -b v0.10.11 mediapipe-v0.10.11 git checkout e135531 sh ./setup_opencv.sh bash ./setup_android_sdk_and_ndk.sh ~/Android/Sdk ~/Android/Ndk r21 [--accept-licenses] bazel build -c opt --config=android_arm64 --linkopt="-s" //mediapipe/tasks/java/com/google/mediapipe/tasks/genai:tasks_genai

Version: mediapipe-0.10.11 TAG: e135531 bazel version 6.1.1 Android Version: 10

[huazhusun]

In funtion "LlmInferenceEngine_CreateSession_Helper", It always check the backend as "CPU", why?

MP_ASSIGN_OR_RETURN(auto backend, model_data->ReadMetadata( mediapipe::tasks::genai::llm_utils::kLlmBackendName)); RET_CHECK_EQ(backend, "cpu"); Does it mean that LlmInferenceEngine only support CPU?

[kuaashish]

Hi @huazhusun,

Could you confirm whether you have successfully installed the MediaPipe framework from the provided source here? have you been able to run the Hello World example as instructed in the same source?

Thank you!!

[schmidt-sebastian]

Hi! Sorry this a new release and we still trying to tie up loose ends. For now, our open source code only allows us to run CPU models, while the Maven package include GPU support. It looks like you are running a GPU model on the CPU path. We hope to have a better story soon.