The input packet MPImage returned by `onResult` after `detectAsync` call is corrupted

[definability]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

No

OS Platform and Distribution

Android 11

Mobile device if the issue happens on mobile device

Redmi Note 8T

Browser and version if the issue happens on browser

No response

Programming Language and version

Kotlin

MediaPipe version

0.10.8

Bazel version

6.1.1

Solution

hand_landmarker

Android Studio, NDK, SDK versions (if issue is related to building in Android environment)

Android Studio 2023.1.1, Android SDK 30, Android NDK 21

Xcode & Tulsi version (if issue is related to building for iOS)

No response

Describe the actual behavior

The resulting input packet contains a horizontally stretched RGBA image with vertical stripes

Describe the expected behaviour

The resulting input packet must contain an identical image to the input one if it was RGBA/RGB

Standalone code/steps you may have used to try to get what you need

• Use https://github.com/googlesamples/mediapipe repository.
• Set up the MediaPipe Tasks Hand Landmark Detection Android Demo.
• Set a breakpoint anywhere within the function returnLivestreamResult.
• Launch debug of the application or landmarkerLiveStreamModeResultsFallsWithinAcceptedRange test.
• Inspect the variable input of type MPImage. Android Studio allows viewing the contents of bitmaps. Go to input->containerMap->[AUtoValueMPImageProperties]->value->bitmap and click View Bitmap.
• After the image is loaded, you should see an image with alternating R, G, B, and "normal" stripes.

Other info / Complete Logs

No response

[definability]

Here is the example image (from the tests from the examples repository)

Here is the resulting input packet:

[definability]

Proposed fix: https://github.com/google/mediapipe/pull/5018.

P.S. Does anybody know how to build a jar library like com.google.mediapipe:tasks-vision to test the changes easily? P.P.S. Is there a way to launch the unit tests https://github.com/google/mediapipe/tree/master/mediapipe/tasks/javatests/com/google/mediapipe/tasks?

[chiu14578]

Any follow solutions?

[Satheeshkummari]

is this fixed?

[definability]

@chiu14578, @Satheeshkummari, I have a fix that has been awaiting review for almost a year: https://github.com/google-ai-edge/mediapipe/pull/5018. I think you can pull the changes and compile the library with Bazel.

[HarrisonUnifyAI]

@definability according to this post the fix looks much easier. Just changing a call from RGB to RGBA https://github.com/google-ai-edge/mediapipe/pull/6046

[HarrisonUnifyAI]

@whhone please look at this issue. It's a critical issue for me

[whhone]

@gkarpiak @schmidt-sebastian please take a look.

[schmidt-sebastian]

Fix pending review internally.

[definability]

@schmidt-sebastian, let me know if you use my PR so I know whether I should fix the conflicts that arose during the two years: https://github.com/google-ai-edge/mediapipe/pull/5018

[definability]

@definability according to this post the fix looks much easier. Just changing a call from RGB to RGBA #6046

@HarrisonUnifyAI, yes, the simple fix provided in https://github.com/google-ai-edge/mediapipe/pull/6046 should fix the particular case for an RGBA input. However, once I got deeper into the code, I saw that AndroidPacketGetter has specific methods for greyscale, RGB, and RGBA images, which were not used for reasons beyond my comprehension. Simply replacing AndroidPacketGetter.getBitmapFromRgb with AndroidPacketGetter.getBitmapFromRgba means breaking backward compatibility for those who rely on RGB input, so I decided to fix the entire flow. One might argue that there are no use cases for greyscale or RGB inputs, but as long as I had time and passion, why not improve the code even for the sake of consistency?

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No

[HarrisonUnifyAI]

@schmidt-sebastian do you by chance know when we can expect a release or is there a nightly build we can pin to maybe?