mlkit icon indicating copy to clipboard operation
mlkit copied to clipboard
verified publisher icon googlesamples

[Bug report] Non-Accurate Pose Detection Crashes In Non-CPU Mode

Open easytarget2000 opened this issue 3 years ago • 5 comments

Describe the bug A custom app using the MLKit library crashes on the first call of Detector<Pose>#process(android.media.Image, int).

Logcat excerpt:

2022-04-07 18:55:06.227 1225-16641/? I/CHIUSECASE: Refill: StreamBufferCache for stream 0x7a1033e1e8 has been abandoned. Can not refill cache any more.
2022-04-07 18:55:06.261 1225-16641/? I/CHIUSECASE: Refill: StreamBufferCache for stream 0x7a1033e1e8 has been abandoned. Can not refill cache any more.
2022-04-07 18:55:06.294 1225-16641/? I/CHIUSECASE: Refill: StreamBufferCache for stream 0x7a1033e1e8 has been abandoned. Can not refill cache any more.
2022-04-07 18:55:06.327 1225-16641/? I/CHIUSECASE: Refill: StreamBufferCache for stream 0x7a1033e1e8 has been abandoned. Can not refill cache any more.
2022-04-07 18:55:06.361 1225-16641/? I/CHIUSECASE: Refill: StreamBufferCache for stream 0x7a1033e1e8 has been abandoned. Can not refill cache any more.
2022-04-07 18:55:06.394 1225-16641/? I/CHIUSECASE: Refill: StreamBufferCache for stream 0x7a1033e1e8 has been abandoned. Can not refill cache any more.
2022-04-07 18:55:06.427 1225-16641/? I/CHIUSECASE: Refill: StreamBufferCache for stream 0x7a1033e1e8 has been abandoned. Can not refill cache any more.
2022-04-07 18:55:06.461 1225-16641/? I/CHIUSECASE: Refill: StreamBufferCache for stream 0x7a1033e1e8 has been abandoned. Can not refill cache any more.
2022-04-07 18:55:06.494 1225-16641/? I/CHIUSECASE: Refill: StreamBufferCache for stream 0x7a1033e1e8 has been abandoned. Can not refill cache any more.
2022-04-07 18:55:06.527 1225-16641/? I/CHIUSECASE: Refill: StreamBufferCache for stream 0x7a1033e1e8 has been abandoned. Can not refill cache any more.
2022-04-07 18:55:06.528 1434-6440/? E/Camera3-OutputUtils: requestStreamBuffers: request too much buffers for stream 0: at HAL: 8 + requesting: 1 > max: 8
2022-04-07 18:55:06.528 1225-16641/? W/CHIUSECASE: RequestStreamBufferFromFwk: request failed due to error: 4
2022-04-07 18:55:06.528 1225-16641/? W/CHIUSECASE: RequestStreamBufferFromFwk: buffer_ret[0] error: 2
2022-04-07 18:55:06.528 1225-16641/? I/CHIUSECASE: Refill: StreamBufferCache for stream 0x7a104346a8 marked as abandoned.
2022-04-07 18:55:06.560 1225-2027/? I/CHIUSECASE: RequestStreamBuffers: dummy buffer returned for stream 0x7a104346a8
2022-04-07 18:55:06.560 1225-16641/? I/CHIUSECASE: Refill: StreamBufferCache for stream 0x7a1033e1e8 has been abandoned. Can not refill cache any more.
2022-04-07 18:55:06.560 1225-16641/? I/CHIUSECASE: Refill: StreamBufferCache for stream 0x7a104346a8 has been abandoned. Can not refill cache any more.
2022-04-07 18:55:06.576 1452-1452/? I/EaselThermalMonitor: Cooling Device: mnh cur_state 0 max_state 4
2022-04-07 18:55:06.594 1225-2029/? I/CHIUSECASE: RequestStreamBuffers: dummy buffer returned for stream 0x7a104346a8
2022-04-07 18:55:06.594 1225-16641/? I/CHIUSECASE: Refill: StreamBufferCache for stream 0x7a1033e1e8 has been abandoned. Can not refill cache any more.
2022-04-07 18:55:06.594 1225-16641/? I/CHIUSECASE: Refill: StreamBufferCache for stream 0x7a104346a8 has been abandoned. Can not refill cache any more.
2022-04-07 18:55:06.616 1434-16606/? E/Camera3-OutputUtils: Camera 0: notifyError: cannot find in-flight request on frame 435 error
2022-04-07 18:55:06.617 16534-16550/eu.ezytarget.exercisedetection E/CameraDevice-JV-0: Receive capture error on unknown request ID -1
2022-04-07 18:55:06.618 1434-16606/? E/Camera3-OutputUtils: Camera 0: notifyError: cannot find in-flight request on frame 437 error
2022-04-07 18:55:06.618 16534-16550/eu.ezytarget.exercisedetection E/CameraDevice-JV-0: Receive capture error on unknown request ID -1
2022-04-07 18:55:06.619 1434-16606/? E/Camera3-OutputUtils: Camera 0: notifyError: cannot find in-flight request on frame 439 error
2022-04-07 18:55:06.620 16534-16550/eu.ezytarget.exercisedetection E/CameraDevice-JV-0: Receive capture error on unknown request ID -1
2022-04-07 18:55:06.621 1434-16606/? E/Camera3-OutputUtils: Camera 0: notifyError: cannot find in-flight request on frame 441 error
2022-04-07 18:55:06.622 16534-16550/eu.ezytarget.exercisedetection E/CameraDevice-JV-0: Receive capture error on unknown request ID -1
2022-04-07 18:55:06.623 1434-16606/? E/Camera3-OutputUtils: Camera 0: notifyError: cannot find in-flight request on frame 443 error
2022-04-07 18:55:06.623 16534-16550/eu.ezytarget.exercisedetection E/CameraDevice-JV-0: Receive capture error on unknown request ID -1
2022-04-07 18:55:06.625 1434-16606/? E/Camera3-OutputUtils: Camera 0: notifyError: cannot find in-flight request on frame 445 error
2022-04-07 18:55:06.625 16534-16550/eu.ezytarget.exercisedetection E/CameraDevice-JV-0: Receive capture error on unknown request ID -1
2022-04-07 18:55:06.626 1434-16606/? E/Camera3-OutputUtils: Camera 0: notifyError: cannot find in-flight request on frame 447 error

To Reproduce Example Steps to reproduce the behavior in sample app:

  1. Create an Android app using the library com.google.mlkit:pose-detection:18.0.0-beta2.
  2. Build the app with compileSdk=32, buildToolsVersion="32.0.0", targetSdk=32.
  3. Initialize a PoseDetector like so:
PoseDetection.getClient(
    PoseDetectorOptions.Builder()
        .setDetectorMode(PoseDetectorOptions.STREAM_MODE)
        .build()
)
  1. Use the PoseDetector to process a media image obtained from a camera2 ImageReader.
  2. Run the app on a Pixel 3 XL running Android 12.
  3. Observe that the app crashes and outputs a seemingly infinite amount of cannot find in-flight request on frame error messages.

Expected behavior The app does not crash.

Note: The app works as expected when either: a. PoseDetectorOptions.Builder is called with setPreferredHardwareConfigs(CPU). b. AccuratePoseDetectorOptions.Builder is used instead and the pose detection is run in accurate mode. This does not require setting the preferred hardware config.

SDK Info:

  • com.google.mlkit:pose-detection:18.0.0-beta2

Smartphone:

  • Device: Pixel 3 XL
  • Device OS: Android 12

Development Environment:

  • IDE Version: Android Studio Bumblebee 2021.1.1 Path 3

easytarget2000 avatar Apr 07 '22 17:04 easytarget2000

Thanks for providing the details. Let's first try to understand whether this issue is more related to the ML Kit detector internal implementation or the camera setup.

Did you try the ML Kit vision quickstart app? Did you get the same error on your device?

ai-plays avatar Apr 07 '22 19:04 ai-plays

I have now tried the Vision Quickstart app with performanceMode == POSE_DETECTOR_PERFORMANCE_MODE_FAST and preferGPU=false. I have noticed the following behaviour both in the Kotlin and Java section of the app, when selecting "Pose Detection" from the dropdown menu:

  • The app completely halts for about 7 seconds. This happened the first two times I selected "Pose Detection" from the menu.
  • After that, the app only halted for slightly under a second when I selected "Pose Detection". Restarting the app and even restarting my device did not change this.

This test was done on a Pixel 3 XL.

I have also tested the Quickstart app and my own app on a Pixel 6. On that device I did not run into the crash that I described in my first post.

My initial conclusions:

  • The initialization of or the first call to PoseDetector takes between half a second and up to multiple seconds.
  • During this phase my app is holding on to camera frames and does not call Image#close. This eventually leads to some sort of timeout which causes a crash.

easytarget2000 avatar Apr 09 '22 09:04 easytarget2000

Hi easytarget2000,

I did a quick check on my Pixel 3XL with Android 12 and the latest vision quickstart app, but wasn't able to reproduce the issue. Let me double check with my co-workers. Before that, to unblock you, you could try setting the preferred hardware to only be CPU or using a previous version of the Pose SDK: 18.0.0-beta1

I will keep you posted as we investigate into this.

ai-plays avatar Apr 11 '22 00:04 ai-plays

Thank you for looking into this. For now, I will go with setting the CPU as preferred hardware.

I have also stumbled upon this comment in CameraSource.java that sounds similar to what I experience:

// Through trial and error it appears that two free buffers, in addition to the two buffers
// used in this code, are needed for the camera to work properly.  Perhaps the camera has
// one thread for acquiring images, and another thread for calling into user code.  If only
// three buffers are used, then the camera will spew thousands of warning messages when
// detection takes a non-trivial amount of time.

I will see if I can apply this workaround to my implementation.

easytarget2000 avatar Apr 11 '22 11:04 easytarget2000