🐛 V3 runAsync doesn't work

[manolo-battista]

What's happening?

On docs is described:

But I think it needs update, cause request frame as first parameter and func (callback) as second parameter.

The callback function is never called. I saw that code reach the func on runOnAsyncContext function, but I think enter always on finally cause I can't see my log ok 2.

Reproduceable Code

No response

Relevant log output

No response

Camera Device

No response

Device

iPhone 14 Pro

VisionCamera Version

3.0.0

Can you reproduce this issue in the VisionCamera Example app?

• [ ] I can reproduce the issue in the VisionCamera Example app.

Additional information

• [ ] I am using Expo
• [X] I have enabled Frame Processors (react-native-worklets-core)
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar issues in this repository and found none.

[abdelrahman-muntaser]

@manolo-battista did you find any solution for this ?

[manolo-battista]

@manolo-battista did you find any solution for this ?

no, unfortunately I still didn't find any solution. For my projects at the moment I'm using runAtTargetFps to run lower FPS rate.

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log("I'm running synchronously at 60 FPS!")
  runAtTargetFps(2, () => {
    'worklet'
    console.log("I'm running synchronously at 2 FPS!")
  })
}, [])

[mrousavy]

Hi. Would've been great if you could've sent a PR for that to fix it in docs.

But in my tests, runAsync works - can you put a console.log in there to double check that it does not get called?

Can you reproduce it in the example app?

[rodgomesc]

hey @mrousavy i think we should keep this issue open, i'm facing the same issue since this function was released, can you provide a minimal example that's working for you?

[rodgomesc]

some weirdness scope things happening

UseCase1:

logcat throws JSI rethrowing JS exception: Regular javascript function cannot be shared. Try decorating the function with the 'worklet' keyword to allow the javascript function to be used as a worklet."

  const frameProcessor = useFrameProcessor(
    (frame) => {
      "worklet";
      runAsync(frame, () => {
        "worklet";
        console.log("my awesome log");
      });
    },
    []
  );
  

UseCase2:

do not throws anything onlogcat but frameContext is always undefined, the callback is not being triggered for some reason

  const myFn = (frameContext) => {
    "worklet";
    console.log("Here", frameContext);
  };

  const frameProcessor = useFrameProcessor(
    (frame) => {
      "worklet";
      runAsync(frame, myFn);
    },
    [myFn]
  );
  

[mrousavy]

Oh! Hm that's weird, I remember this worked when I built it.

@chrfalch do you maybe have any insights?

[rodgomesc]

oh wait

it seems that by design the frame it's not being passed to myFn as a callback prop, this forces us to uses arrow functions to hook the frame scope, also it seems that's no guarantees that internal.decrementRefCount() is being called after my long running function finishes the execution?, which causes a java.lang.IllegalStateException: Image is already closed,

https://github.com/mrousavy/react-native-vision-camera/blob/764897dcf119e26668b9fb6c6eb78f0e2993ffd1/package/src/FrameProcessorPlugins.ts#L50-L55

[mrousavy]

Wait what? not sure if I follow -

it seems that by design the frame it's not being passed to myFn as a callback prop

no you're right, you can just use the Frame inside your lambda directly, the lambda has no parameters.

there's no guarantees that internal.decrementRefCount() is being called after my long running function finishes the execution

Why not? It runs in finally, no?

[rodgomesc]

@mrousavy yep you are totally right, i was very drunk that day sorry

found a pattern that doesn't make sense for me, example app works my app doesn't work, tried different combination of libs and now i'm using same version that's on example but it always end up with

[rodgomesc]

i have a serie of frame processors at this moment running in sequence

something like this

const frameProcessor = (frame) => {
    'worklet';

     // processors
     const faces = faceDetectorProcessor(frame)
     const luminanceResults = luminanceDetectionProcessor(frame, faces)
     const spoofResults = antiSpoofingProcessor(frame);


    // callbacks
    handleSpoofingDetection(spoofResults)
    handleLuminanceDetection(luminanceResults)

},[handleSpoofingDetection, handleLuminanceDetection])

the problem is that at this point the frame are being deallocated before i finish my operations, so i'm thinking to implement something like this, to drop some frames while my processors still running

const frameProcessor = (frame) => {
    'worklet';

    // just drop the frame
    if(!jsiUtil.finishedOperations) return;  

    const frameCopy = jsiUtil.copyFrame(frame);

     // processors      
     const faces = faceDetectorProcessor(frameCopy)
     const luminanceResults = luminanceDetectionProcessor(frameCopy, faces)
     const spoofResults = antiSpoofingProcessor(frameCopy);

    // cleanup frame copy
    jsiUtil.ReleaseFrameCopy()    

    // callbacks
    handleSpoofingDetection(spoofResults)
    handleLuminanceDetection(luminanceResults)

},[handleSpoofingDetection, handleLuminanceDetection])

would be nice to hear your opinion on this temporary solution to make sure i'm not missing anything

some final notes:

i'm linking my lib with visionCamera.so and unwraping the frame, i've tried increment refcount before my operations and decrement on the last one, but it seems to be ignored need further investigation

[mrousavy]

Woah, that's a weird error. Can you add console.log statements to your code (begin and end of sync calls, and begin and end of async calls) and then also to the native Frame's release method?

This is weird and you shouldn't need such workarounds - it should just work magically with ref counting.

[rodgomesc]

i still couldn't figure out what's happening, i'll try to upload something reproducible

[rodgomesc]

Haaa I found out, i have a monorepo with this structure

-- packages ---- playground ---- mylib

if i have reanimated in my lib i face this issue, removed reanimated from mylib and now i'm using it only in the playground, working like a charm

Edit1:

if i try a simple console.log after removing reanimated lib it works

useFrameProcessor(frame => .....

 runAsync(frame, () => {
 'worklet'
    console.log('hello from runAsync')
  })
  
},[])

if i try to use any function inside of it, it crashes exactly with same error as before

 const myFunc = () => {
    'worklet'
     console.log('hello from runAsync')
 }
 
useFrameProcessor(frame => .....

 runAsync(frame, () => {
  'worklet'
    myFunc();
  })
  
},[myFunc])
 
 

[rodgomesc]

@mrousavy fyi it's reproducible on example app now

[mrousavy]

Wait so Reanimated makes it break? Yea it's a bit weird that Worklets and reanimated do the same thing... Maybe we can fix this compatibility for now and think about a better solution in the future.

[rodgomesc]

Wait so Reanimated makes it break?

that's what i thought because after removing reanimated i can do a console.log inside the runAsync, however if i call any function inside it i got the same error

[gbark]

Similar issue here. Using @ismaelmoreiraa/vision-camera-ocr, any call to scanOCR using runAsync throws:

JSI rethrowing JS exception: Exception in HostFunction: java.lang.ClassNotFoundException: Didn't find class "com.facebook.jni.MapIteratorHelper" on path: DexPathList[[directory "."],nativeLibraryDirectories=[/system/lib64, /system/system_ext/lib64, /system/lib64, /system/system_ext/lib64]]

Error: Exception in HostFunction: java.lang.ClassNotFoundException: Didn't find class "com.facebook.jni.MapIteratorHelper" on path: DexPathList[[directory "."],nativeLibraryDirectories=[/system/lib64, /system/system_ext/lib64, /system/lib64, /system/system_ext/lib64]]
    at call (native)
    at scanOCR (/Users/username/dev/app/node_modules/@ismaelmoreiraa/vision-camera-ocr/src/index.tsx:8:21)
    at fn (native)
    at anonymous (/Users/username/dev/app/packages/app/src/features/foo.tsx:18:27)
    at fn (native)
    at anonymous (/Users/username/dev/app/node_modules/react-native-vision-camera/src/FrameProcessorPlugins.ts:6:9)

Running the same exact same code using runAtTargetFps works fine.

I'm not using Reanimated in my project.

[mrousavy]

Btw.; here's an explanation and a fix: https://github.com/margelo/react-native-worklets-core/issues/136

[mrousavy]

This is not related to VisionCamera, but rather about Reanimated.

If you do not use Reanimated, runAsync works fine. If you do use Reanimated, you need to enable processNestedWorklets in the Reanimated's babel plugin. See https://github.com/software-mansion/react-native-reanimated/issues/5576 for more info

[Cosmorider]

Enabling processNestedWorklets solved the error for me, but now this error pops up: Frame Processor Error: Value is undefined, expected an Object, js engine: VisionCamera

This is my code (from react-native-fast-tflite), note that the model is printed out, but then it shows the error, I'm trying to make the model not interfere with how the camera output is displayed in my app (it interferes on Android but not iOS):

  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
  
    const data = resize(frame, {
      size: {
        // center-crop
        x: (frame.width / 2) - (320 / 2),
        y: (frame.height / 2) - (320 / 2),
        width: 320,
        height: 320,
      },
      pixelFormat: 'rgb',
      dataType: 'uint8'
    })

    runAsync(frame, () => {
      'worklet'
      console.log(model)

      const output = model.runSync([data])
    
      const numDetections = output[0]
    })
  }, [model])

If I pass the frame into runAsync, I get this error: ERROR Frame Processor Error: Exception in HostFunction: no ArrayBuffer attached, js engine: VisionCamera

Not sure if I'm misunderstanding some fundamentals here.

"react-native-vision-camera": "^3.8.2",
"react-native-worklets-core": "^0.2.4",
"react-native-reanimated": "^3.6.1",

[mrousavy]

ArrayBuffers can't be shared (right @chrfalch?), do the resize also in runAsynx.

[chrfalch]

Correct.

[Cosmorider]

Sorry, I meant that I passed the resize into runAsync and then I got this error, if that was what you were asking for: ERROR Frame Processor Error: Exception in HostFunction: no ArrayBuffer attached, js engine: VisionCamera

[mrousavy]

Yeah - try to move the resize call also into runAsync.

[Cosmorider]

I did that, and I get this error: ERROR Frame Processor Error: Exception in HostFunction: java.lang.ClassNotFoundException: Didn't find class "com.mrousavy.camera.frameprocessor.Frame" on path: DexPathList[[directory "."],nativeLibraryDirectories=[/system/lib64, /system/product/lib64, /system/lib64, /system/product/lib64]], js engine: VisionCamera

When I use this code:

  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'

    runAsync(frame, () => {
      'worklet'

      const data = resize(frame, {
        size: {
          // center-crop
          x: (frame.width / 2) - (320 / 2),
          y: (frame.height / 2) - (320 / 2),
          width: 320,
          height: 320,
        },
        pixelFormat: 'rgb',
        dataType: 'uint8'
      })
    })
  }, [])

Edit: If I have both resize outside and inside runAsync, it just crashes with no symbolic stack trace:

const frameProcessor = useFrameProcessor((frame) => {
    'worklet'

    const data = resize(frame, {
      size: {
        // center-crop
        x: (frame.width / 2) - (320 / 2),
        y: (frame.height / 2) - (320 / 2),
        width: 320,
        height: 320,
      },
      pixelFormat: 'rgb',
      dataType: 'uint8'
    })

    runAsync(frame, () => {
      'worklet'

      const data = resize(frame, {
        size: {
          // center-crop
          x: (frame.width / 2) - (320 / 2),
          y: (frame.height / 2) - (320 / 2),
          width: 320,
          height: 320,
        },
        pixelFormat: 'rgb',
        dataType: 'uint8'
      })
    })
  }, [])

[rodgomesc]

I did that, and I get this error: ERROR Frame Processor Error: Exception in HostFunction: java.lang.ClassNotFoundException: Didn't find class "com.mrousavy.camera.frameprocessor.Frame" on path: DexPathList[[directory "."],nativeLibraryDirectories=[/system/lib64, /system/product/lib64, /system/lib64, /system/product/lib64]], js engine: VisionCamera

that so weird, same problem here

[mrousavy]

From reading the error message, it looks like the reason the JNI class cannot be found in runAsync is because there is no JNI Environment set up on that specific Thread - which is weird, because we set it up here: https://github.com/margelo/react-native-worklets-core/blob/d8dae58ffac6b7050bb0b410b69acc621dcf74d8/cpp/WKTJsiWorkletContext.cpp#L147-L149

Is that not called when invoking a worklet from JS, @chrfalch ?

[mrousavy]

Hey @rodgomesc and @Cosmorider - a discord user posted a SIGABRT stacktrace with symbols attached so I was able to figure out that this came from JVisionCameraScheduler - can you guys maybe try if this PR fixes the issue for you? https://github.com/mrousavy/react-native-vision-camera/pull/2457

[MSchmidt]

I'm not seeing the same error. I have this one when using runAsync:

Frame Processor Error: Exception in HostFunction: java.lang.ClassNotFoundException: Didn't find class "com.facebook.jni.MapIteratorHelper" on path: DexPathList[[directory "."],nativeLibraryDirectories=[/system/lib64, /system_ext/lib64, /system/lib64, /system_ext/lib64]], js engine: VisionCamera

The new PR does not fix this particular one.

[mrousavy]

@MSchmidt can you try to add a

facebook::jni::ThreadScope scope;

right at the top of this method: https://github.com/mrousavy/react-native-vision-camera/blob/02bc8a979c192707efc5d6e1f424812e36f6369f/package/android/src/main/cpp/frameprocessor/java-bindings/JFrameProcessor.cpp#L35-L45

? Let me know if that works for you