Why does the mediapipe not use tflite runtime for genai?

[Hozzu]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

No

OS Platform and Distribution

Ubuntu 22.04

MediaPipe Tasks SDK version

No response

Task name (e.g. Image classification, Gesture recognition etc.)

GenAI

Programming Language and version (e.g. C++, Python, Java)

C++

Describe the actual behavior

Mediapipe GenAI task uses tflite runtime

Describe the expected behaviour

Mediapipe GenAI task does not use tflite runtime

Standalone code/steps you may have used to try to get what you need

I saw the mediapipe task code for genai and found that it does not use tflite runtime.
It seems to only use tflite file format and independent runtime to run it.
The independent runtime only supports CPU now.

However, the tflite runtime supports GPU via OpenCL and Hexagon delegate, etc.
Why does the mediapipe not use tflite runtime for genai?
Could you explain the reason?

Other info / Complete Logs

No response

[kuaashish]

Hi @Hozzu,

Currently, our GenAI Task C API only supports the CPU. We have acknowledged the need for GPU support, feature request https://github.com/google-ai-edge/mediapipe/issues/5305 and are working to implement it. In the meantime, if you need GPU support, it is available through our Maven package.

Thank you!!

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[jason6q]

@kuaashish, is there an ETA on when there will be GPU support for next mediapipe version?

Thanks for the updates!

[kuaashish]

Hi @nu1ptr,

Currently, we do not have an ETA for this. Based on the update received from our team, it will not be available soon. However, we are marking this issue as a feature request and sharing it with the team so it will be on our radar.

Thank you!!

[yichunk]

Hi, there is no exact ETA when the GPU TFLite runtime will be ready for GenAI. It involves KV cache updates which requires to update from TFLite runtime as well.