🐛 Crash on frame.toArrayBuffer()

[jslok]

What's happening?

I am trying to setup a frame processor with tflite as the docs show but I'm getting a crash. https://github.com/mrousavy/react-native-fast-tflite This line appears to be the issue: const data = frame.toArrayBuffer()

Reproduceable Code

(from the docs)
const model = useTensorflowModel(require('object_detection.tflite'))

const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    if (model.state !== "loaded") return

    const data = frame.toArrayBuffer()
    // do RGB conversion if the Frame is not already in RGB Format
    const outputs = model.model.runSync([data])

    const detection_boxes = outputs[0]
    const detection_classes = outputs[1]
    const detection_scores = outputs[2]
    const num_detections = outputs[3]
    console.log(`Detected ${num_detections[0]} objects!`)

    for (let i = 0; i < detection_boxes.length; i += 4) {
        const confidence = detection_scores[i / 4]
        if (confidence > 0.7) {
            // Draw a red box around the object!
            const left = detection_boxes[i]
            const top = detection_boxes[i + 1]
            const right = detection_boxes[i + 2]
            const bottom = detection_boxes[i + 3]
            const rect = SkRect.Make(left, top, right, bottom)
            frame.drawRect(rect, SkColors.Red)
        }
    }
}, [model])

return (
    <Camera frameProcessor={frameProcessor} {...otherProps} />
)

Relevant log output

01-09 19:06:41.740  1469  1469 I CameraService: CameraService::connect call (PID 24871 "com.rn73scanner", camera ID 0) and Camera API version 2
01-09 19:06:41.742  1469  1469 I Camera2ClientBase: Camera 0: Opened. Client: com.rn73scanner (PID 24871, UID 11568)
01-09 19:06:41.759  3458  3560 D LyricConfigProviderService: onCameraOpened: cameraId 0 used by packageId com.rn73scanner for versionCode 1
01-09 19:06:42.286   771  3852 W gralloc4: Unable to set buffer name SurfaceView[com.rn73scanner/com.rn73scanner.MainActivity]#2(BLAST Consumer)2: File name too long
01-09 19:06:42.454   771   771 W gralloc4: Unable to set buffer name SurfaceView[com.rn73scanner/com.rn73scanner.MainActivity]#2(BLAST Consumer)2: File name too long
01-09 19:06:42.536   771   771 W gralloc4: Unable to set buffer name SurfaceView[com.rn73scanner/com.rn73scanner.MainActivity]#2(BLAST Consumer)2: File name too long
01-09 19:06:42.644 24871 24885 F libc    : Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x3b8000003b800004 in tid 24885 (binder:24871_2), pid 24871 (com.rn73scanner)
01-09 19:06:42.913 25209 25209 F DEBUG   : Cmdline: com.rn73scanner
01-09 19:06:42.913 25209 25209 F DEBUG   : pid: 24871, tid: 24885, name: binder:24871_2  >>> com.rn73scanner <<<
01-09 19:06:42.931  1733 25212 W ActivityTaskManager:   Force finishing activity com.rn73scanner/.MainActivity
01-09 19:06:42.990  1733  3827 I ActivityManager: Process com.rn73scanner (pid 24871) has died: fg  TOP
01-09 19:06:42.991  1733  6806 I ImeTracker: com.rn73scanner:8e69209d: onRequestHide at ORIGIN_SERVER_HIDE_INPUT reason HIDE_REMOVE_CLIENT
01-09 19:06:42.992  1733  6806 I ImeTracker: com.rn73scanner:8e69209d: onCancelled at PHASE_SERVER_SHOULD_HIDE
01-09 19:06:42.992  1733  3828 I WindowManager: WIN DEATH: Window{540da3d u0 com.rn73scanner/com.rn73scanner.MainActivity}
01-09 19:06:42.992  1733  3828 W InputManager-JNI: Input channel object '540da3d com.rn73scanner/com.rn73scanner.MainActivity (client)' was disposed without first being removed with the input manager!
01-09 19:06:43.022  1733  1895 W WindowManager: Failed to deliver inset control state change to w=Window{540da3d u0 com.rn73scanner/com.rn73scanner.MainActivity EXITING}
01-09 19:06:43.044  1733  1890 V WindowManager:     info={id=1757 t=CLOSE f=0x10 trk=0 r=[0@Point(0, 0)] c=[{WCT{RemoteToken{ab99297 Task{2bd7b0 #1 type=home}}} m=TO_FRONT f=SHOW_WALLPAPER|MOVE_TO_TOP leash=Surface(name=Task=1)/@0x5b11076 sb=Rect(0, 0 - 1080, 2400) eb=Rect(0, 0 - 1080, 2400) d=0},{WCT{RemoteToken{78b6715 Task{88d2c21 #8562 type=standard A=11568:com.rn73scanner}}} m=CLOSE f=NONE leash=Surface(name=Task=8562)/@0x83e3e59 sb=Rect(0, 0 - 1080, 2400) eb=Rect(0, 0 - 1080, 2400) d=0},{null m=TO_FRONT f=IS_WALLPAPER leash=Surface(name=WallpaperWindowToken{28529bb token=android.os.Binder@a72c64a})/@0xceb595a sb=Rect(0, 0 - 1080, 2400) eb=Rect(0, 0 - 1080, 2400) d=0}]}
01-09 19:06:43.065  1733  1895 W WindowManager: Failed to deliver inset control state change to w=Window{540da3d u0 com.rn73scanner/com.rn73scanner.MainActivity EXITING}
01-09 19:06:43.119  1469 25039 I Camera2ClientBase: ~Camera2ClientBase: Client object's dtor for Camera Id 0 completed. Client was: com.rn73scanner (PID 24871, UID 11568)
01-09 19:06:43.248  3163  3177 W ndroid.systemui: ApkAssets: Deleting an ApkAssets object '<empty> and /data/app/~~XXZsiDKy1LR0BQzGoVGCDg==/com.rn73scanner-32195I_3Guot2jNpBHoMGA==/base.apk' with 1 weak references
01-09 19:06:43.282  1733  6806 W WindowManager: Exception thrown during dispatchAppVisibility Window{540da3d u0 com.rn73scanner/com.rn73scanner.MainActivity EXITING}
01-09 19:06:43.434  1733  1894 W ActivityTaskManager: Activity top resumed state loss timeout for ActivityRecord{5855a88 u0 com.rn73scanner/.MainActivity t-1 f}}

Camera Device

{"autoFocusSystem": "contrast-detection", "fieldOfView": 83.97117848314457, "maxFps": 60, "maxISO": 11906, "maxZoom": 8, "minFps": 1, "minISO": 47, "photoHeight": 3024, "photoWidth": 4032, "pixelFormats": ["yuv", "native"], "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": true, "videoHeight": 1080, "videoStabilizationModes": ["off", "standard", "cinematic", "off", "cinematic-extended"], "videoWidth": 1920}

Device

Pixel 7

VisionCamera Version

3.7.0

Can you reproduce this issue in the VisionCamera Example app?

I cannot run the example app but I encountered this issue in a fresh install with rn 0.73.0.

Additional information

• [ ] I am using Expo
• [X] I have enabled Frame Processors (react-native-worklets-core)
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar issues in this repository and found none.

[jslok]

3.6.16 appears to not have this issue.

[mrousavy]

Hm, I didn't change anything from 3.6.16 to 3.7.0 regarding toArrayBuffer.

Can you play around with this line here maybe? https://github.com/mrousavy/react-native-vision-camera/blob/2b106225590e2694f25a083003bc983d2cb490bb/package/android/src/main/java/com/mrousavy/camera/core/VideoPipeline.kt#L100

Maybe you need to set this to HardwareBuffer.USAGE_GPU_SAMPLED_IMAGE or HardwareBuffer.USAGE_CPU_READ_OFTEN.

[jslok]

I see this issue on Android simulator as well. Tested and seen on android 12 and 13 on simulator, 14 on real device.

Tried many other values for usage but same crash.

[mrousavy]

Tried many other values for usage but same crash

Which ones exactly? 😄

[jslok]

USAGE_COMPOSER_OVERLAY USAGE_CPU_WRITE_RARELY USAGE_CPU_WRITE_OFTEN USAGE_GPU_COLOR_OUTPUT USAGE_GPU_MIPMAP_COMPLETE USAGE_GPU_DATA_BUFFER USAGE_GPU_SAMPLED_IMAGE (default)

In all of the above, the app just closes immediately with no error.

USAGE_SENSOR_DIRECT_DATA - screen freezes some seconds before closing

Just tested again on new version 3.8.0 and the issue still exists.

[mrousavy]

Fixed in https://github.com/mrousavy/react-native-vision-camera/pull/2408! :)

[jslok]

Unfortunately still seeing this crash with 3.8.1 on toArrayBuffer.

I also tried the new vision-camera-resize-plugin (thanks!) with react-native-fast-tflite and getting some other errors which I believe are not related so I will open a new issue for that.

[moghtader]

Same - seeing issues with the .toArrayBuffer() on the latest. It always ends in a segfault.

A resource failed to call Surface.release. 2024-01-18 17:48:27.083 15800-15812 System com.app W A resource failed to call Surface.release. 2024-01-18 17:48:27.084 15800-15812 System com.app W A resource failed to call HardwareBuffer.close. 2024-01-18 17:48:27.084 15800-15812 System com.app W A resource failed to call HardwareBuffer.close. 2024-01-18 17:48:27.084 15800-15812 System com.app W A resource failed to call HardwareBuffer.close. 2024-01-18 17:48:27.084 15800-15812 System com.app W A resource failed to call HardwareBuffer.close. 2024-01-18 17:48:27.084 15800-15812 System com.app W A resource failed to call HardwareBuffer.close. 2024-01-18 17:48:27.085 15800-15812 System com.app W A resource failed to call HardwareBuffer.close. 2024-01-18 17:48:27.085 15800-15812 System com.app W A resource failed to call HardwareBuffer.close. 2024-01-18 17:48:27.085 15800-15812 System com.app W A resource failed to call HardwareBuffer.close. 2024-01-18 17:48:27.085 15800-15812 libc com.app A Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 in tid 15812 (FinalizerDaemon), pid 15800 ()

[mrousavy]

Are you using pixelFormat="yuv"?