google-ai-edge
Flutter Package?
Hello, just learned about MediaPipe today! It's amazing how many developer resources are open-source. It's also crazy to realize that Google has empowered so much of it. With that being said, I was surprised to see the MediaPipe team has not created a package for Flutter developers since both were founded by Google and the MediaPipe is marketed as cross-platform
System information
- MediaPipe Solution: All solutions available on iOS and Android
- Programming language: Dart
- Are you willing to contribute it: Yes, but my team cannot prioritize this over other tasks.
Will this change the current API?
No. Creating a Flutter package will extend and merge the iOS/Android API such that the new, extended API can be used natively in the Dart language. Nothing about the current API will need to be changed as far as I am concerned.
Who will benefit with this feature?
- Developers and teams using the Flutter framework.
- Developers who want to build application to multiple platforms with one codebase
- Developers who want to do great things with great Google products
- Myself 🫶🏼
Hi @charlieforward9, Thanks for raising this feature request. This looks like an interesting request. We will discuss this feature internally and prioritize it among others.
I have integrated mediapipe into my flutter app but it was really cumbersome. It would be great at least have C api that can be called directly from dart, instead of channel communication separate for each platform.
There is https://github.com/homuler/MediaPipeUnityPlugin/tree/master/mediapipe_api, originally created for Unity but could be repurposed. The problem is - it's not pure C, so generating ffi bindings is not so straightforward, and it is not official.
@szotp-lc
It would be great at least have C api that can be called directly from dart, instead of channel communication separate for each platform.
If it were done this way, would it work on all the mobile and desktop platforms that Flutter supports? If so, then the only separate implementation would be the web? Asking to learn. Thanks
Hi! We are having some discussion about this, but cannot provide a timeline at this point. We certainly recognize that better Flutter support is very important to large part of our community. We will provide updates if and when we can.
@szotp-lc do you mind emailing me at [email protected], I would appreciate hearing more about how you integrated into your Flutter app (unless you didn't mind keeping the conversation in this thread). Thank you!
Honestly I would recommend waiting for official support or using a different library (like ML Kit). It has been a painful process.
Honestly I would recommend waiting for official support or using a different library (like ML Kit).
Official Support > .tflite embedding > MLKit MethodChannel
I built a PoseDetection MethodChannel for iOS using MLKit to compare it with my existing pose_detection_full.tflite file embedding using flutter-tflite. The MethodChannel inference delivered slower than the .tflite file embedding with my implementation.
I am sharing this to show findings, save other people time, and keep this relevant.
Goal
Actual (Embedded .tflite)
https://github.com/google/mediapipe/assets/62311337/26d8be7b-6db4-423e-b54e-04dc3144cc29
Actual (iOS MLKit Method Channel)
https://github.com/google/mediapipe/assets/62311337/b6f09123-1983-497d-8b21-65572874ee2e
My implementation uses Dart Isolates to keep the model processing off of the main thread.
Thanks to Google and their Mediapipe team, we are now able to witness development of flutter-mediapipe. It's great to follow along and see how Google would build a package from the ground up.
I have become a watcher of the repository to get a firsthand sight of the progress and changes, so if anyone else lands here and is interested in the progress of the package. I recommend doing the same.
so as of now is it possible to have a mobile application that is developed using flutter that has mediapipe ?
Yes, there is just a lot of manual setup. I recommend using this sample repo to help get started with it.
I didn't need Android config. If you manage to get it up and running with all updated dependencies, feel free to create a PR to Open Source it.. I am waiting for the original creator @Jaeheee to merge it, so we can keep my fork current till it merges.
Happy Coding!
As a humbled developer who requested this over a year ago. I have subscribed to the construction and have been nothing short of impressed.
I am particularly excited to see Pose Detection begin its development as an included feature at https://github.com/google/flutter-mediapipe/issues/51. It was the Mediapipe solution I was having trouble running performantly that prompted me to open this issue.
Thank you Google AI
@charlieforward9 I have been implementing mediapipe-pose using JS but now I want my application to built for multiple platform on one codebase so I have been trying to migrate to flutter, just wanted to ask whether using platform channel solve that problem as of now or is there some alternate
You can see the performance of Method Channels above. @iarshguleria
I am really hoping for the official repo to perform closer to the "goal".
I am getting started on the vision package for the team.
This is my first large scale contribution, so I cannot provide a timeline at the time. However, I will continue to update as I proceed.
https://github.com/google/flutter-mediapipe/discussions/62
Method Channel doesn't seem to have that great performance as compared to the native code
Hi @charlieforward9,
The official Flutter plugin, Flutter Mediapipe, is now available, and the feature has been added. Please let us know if we can consider this issue resolved and close it internally. If you encounter any further issues while using the Flutter Mediapipe plugin, we recommend opening a new issue in the respective repository. We would be happy to assist you with any further concerns.
Thank you!!
Thank you, @kuaashish, for following up. The scope of this issue has not been fully addressed yet. As outlined above, I am requesting that the vision package be made available through Flutter, which remains incomplete.
I’ve been in contact with @craiglabenz regarding contributing to this effort, and my team member, @DrBubbles42, has started preparing to assist. However, there is currently no timeline for achieving the desired functionality. Raising the priority of building out the vision task internally would be incredibly helpful for us.
I was also looking into using flutter-mediapipe, however, there was no pose detection made available, and I am not sure this plugin is still being actively maintained. Is there any other Mediapipe solution for pose that can be used with Flutter?
Thank you.
Hi @charlieforward9,
The official Flutter plugin, Flutter Mediapipe, is now available, and the feature has been added. Please let us know if we can consider this issue resolved and close it internally. If you encounter any further issues while using the Flutter Mediapipe plugin, we recommend opening a new issue in the respective repository. We would be happy to assist you with any further concerns.
Thank you!!
@richardhfj See https://github.com/google-ai-edge/mediapipe/issues/3922#issuecomment-1972492798
@richardhfj See #3922 (comment)
Thank you for the followup. I looked into this, but I see that tflite helper is now deprecated so I am not exactly sure how to use this.
@richardhfj it still works, just not getting new updates, I use it in my projects while I wait for the new package. If youre a real G, build the vision package from the ground up in flutter_mediapipe and use that! I need it too but am too overemployed to get it done rn.
@richardhfj it still works, just not getting new updates, I use it in my projects while I wait for the new package. If youre a real G, build the vision package from the ground up in
flutter_mediapipeand use that! I need it too but am too overemployed to get it done rn.
Unfortunately not a real G. Not sure where to even begin to build the vision package. I will check out the current package again, thanks.