mrousavy
🐛 Camera preview blank screen when using Skia Frame Processors
What's happening?
Hi @mrousavy, thank you for the great work with this library.
However, I have been having issues with applying filters using useSkiaFrameProcessors, even when using the boilerplate provided in the docs. I've tried two implementations
- My own app using a grain filter (right)
- Example app with an inverted colour filter (left)
The camera preview ends up being just the filter, without any image. Please refer to the code and logs from both cases below.
I'm wondering whether this might be a hardware issue? I'm currently developing on an old Huawei P10 (released in 2017, Android 9). Ive had a friend run the app on a Samsung S23 Ultra and he did not encounter this issue.
Reproduceable Code
// RECIPICS Code
const grainFilter = Skia.RuntimeEffect.Make(`
uniform shader image;
half4 main(vec2 pos) {
vec4 color = image.eval(pos);
float grain = fract(sin(dot(pos, vec2(12.9898, 78.233))) * 43758.5453);
color.rgb += grain * ${color};
return color; }`)
const shaderBuilder = Skia.RuntimeShaderBuilder(grainFilter)
const imageFilter = Skia.ImageFilter.MakeRuntimeShader(shaderBuilder, null, null)
const paint = Skia.Paint()
paint.setImageFilter(imageFilter)
const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
frame.render(paint)
}, [paint])
<Camera
style={{
flex: 1
}}
ref={camera}
style={{ width: 400, height: 400 }}
device={device}
isActive={true}
frameProcessor={frameProcessor}
pixelFormat="yuv"
photo={false}
format={format}
/>
}
// Example App Code
const invertColorsFilter = Skia.RuntimeEffect.Make(`
uniform shader image;
half4 main(vec2 pos) {
vec4 color = image.eval(pos);
return vec4((1.0 - color).rgb, 1.0);
}
`)
const shaderBuilder = Skia.RuntimeShaderBuilder(invertColorsFilter);
const imageFilter = Skia.ImageFilter.MakeRuntimeShader(shaderBuilder, null, null)
const paint = Skia.Paint()
paint.setImageFilter(imageFilter)
const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
frame.render(paint)
}, [paint])
Relevant log output
2024-06-04 17:21:01.877 30369-30668 CameraView com.recipics I invokeOnAverageFpsChanged(11.42263759086189)
2024-06-04 17:21:01.921 30369-30680 GRALLOC com.recipics I LockFlexLayout: baseFormat: 11, yStride: 640, ySize: 307200, uOffset: 307200, uStride: 640
2024-06-04 17:21:01.952 30369-30369 GLConsumer com.recipics E [SurfaceTexture-0-30369-2] checkAndUpdateEglState: invalid current EGLContext
2024-06-04 17:21:01.952 30369-30369 surfaceTexture com.recipics E Exception updateTexImage
java.lang.IllegalStateException: Unable to update texture contents (see logcat for details)
at android.graphics.SurfaceTexture.nativeUpdateTexImage(Native Method)
at android.graphics.SurfaceTexture.updateTexImage(SurfaceTexture.java:249)
at com.shopify.reactnative.skia.PlatformContext.notifyDrawLoop(Native Method)
at com.shopify.reactnative.skia.PlatformContext.-$$Nest$mnotifyDrawLoop(Unknown Source:0)
at com.shopify.reactnative.skia.PlatformContext$1.doFrame(PlatformContext.java:61)
at android.view.Choreographer$CallbackRecord.run(Choreographer.java:1090)
at android.view.Choreographer.doCallbacks(Choreographer.java:893)
at android.view.Choreographer.doFrame(Choreographer.java:809)
at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:1078)
at android.os.Handler.handleCallback(Handler.java:907)
at android.os.Handler.dispatchMessage(Handler.java:105)
at android.os.Looper.loop(Looper.java:216)
at android.app.ActivityThread.main(ActivityThread.java:7625)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:524)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:987)
2024-06-04 16:49:16.970 8524-8776 CameraView com.mrousavy.camera.example I invokeOnAverageFpsChanged(4.060913705583756)
2024-06-04 16:49:17.091 8524-8863 GRALLOC com.mrousavy.camera.example I LockFlexLayout: baseFormat: 11, yStride: 3840, ySize: 8294400, uOffset: 8294400, uStride: 3840
2024-06-04 16:49:17.123 8524-8609 <no-tag> com.mrousavy.camera.example E [ZeroHung]zrhung_get_config: Get config failed for wp[0x0102]
2024-06-04 16:49:17.184 8524-8524 GLConsumer com.mrousavy.camera.example E [SurfaceTexture-0-8524-1] checkAndUpdateEglState: invalid current EGLContext
2024-06-04 16:49:17.185 8524-8524 surfaceTexture com.mrousavy.camera.example E Exception updateTexImage
java.lang.IllegalStateException: Unable to update texture contents (see logcat for details)
at android.graphics.SurfaceTexture.nativeUpdateTexImage(Native Method)
at android.graphics.SurfaceTexture.updateTexImage(SurfaceTexture.java:249)
at com.shopify.reactnative.skia.PlatformContext.notifyDrawLoop(Native Method)
at com.shopify.reactnative.skia.PlatformContext.-$$Nest$mnotifyDrawLoop(Unknown Source:0)
at com.shopify.reactnative.skia.PlatformContext$1.doFrame(PlatformContext.java:61)
at android.view.Choreographer$CallbackRecord.run(Choreographer.java:1090)
at android.view.Choreographer.doCallbacks(Choreographer.java:893)
at android.view.Choreographer.doFrame(Choreographer.java:809)
at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:1078)
at android.os.Handler.handleCallback(Handler.java:907)
at android.os.Handler.dispatchMessage(Handler.java:105)
at android.os.Looper.loop(Looper.java:216)
at android.app.ActivityThread.main(ActivityThread.java:7625)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:524)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:987)
Camera Device
[
{
"sensorOrientation": "landscape-left",
"hardwareLevel": "limited",
"maxZoom": 6,
"minZoom": 1,
"maxExposure": 4,
"supportsLowLightBoost": false,
"neutralZoom": 1,
"physicalDevices": [
"wide-angle-camera"
],
"supportsFocus": true,
"supportsRawCapture": false,
"isMultiCam": false,
"minFocusDistance": 10,
"minExposure": -4,
"name": "0 (BACK) androidx.camera.camera2",
"hasFlash": true,
"hasTorch": true,
"position": "back",
"id": "0"
},
{
"sensorOrientation": "landscape-right",
"hardwareLevel": "limited",
"maxZoom": 6,
"minZoom": 1,
"maxExposure": 4,
"supportsLowLightBoost": false,
"neutralZoom": 1,
"physicalDevices": [
"wide-angle-camera"
],
"supportsFocus": true,
"supportsRawCapture": false,
"isMultiCam": false,
"minFocusDistance": 0,
"minExposure": -4,
"name": "1 (FRONT) androidx.camera.camera2",
"hasFlash": false,
"hasTorch": false,
"position": "front",
"id": "1"
}
]
Device
Huawei P10 (Android 9.0)
VisionCamera Version
4.0.5
Can you reproduce this issue in the VisionCamera Example app?
Yes, I can reproduce the same issue in the Example app here
Additional information
- [ ] I am using Expo
- [X] I have enabled Frame Processors (react-native-worklets-core)
- [X] I have read the Troubleshooting Guide
- [X] I agree to follow this project's Code of Conduct
- [X] I searched for similar issues in this repository and found none.
Can you try setting your pixelFormat to rgb or yuv to see if there's a difference?
Trying in the Example App pixelFormat to yuv produces similar results
however rgb turns the filter grey, as seen here
Yup, it does render, but theres definitely some overhead due to the frameprocessor.
Then it surely is the paint (or the shader) that's wrong. I think this is more of a question than a VisionCamera bug report.
Overhead; yea I can see that it only runs at 3 FPS on this Android phone - is that an old phone? @wcandillon I think this is something we might need to investigate - it's either the makeNonTextureImage(), or our makeImageFromPlatformBuffer(...) function that's really slow here... 🤔
I would like to check the performance of the makeImageFromPlatformBuffer function on Android. I would be interesting to know about the phone being used here as well as parameters that would allow me to reproduce the issue (e.g a video file that has the same properties).
Hi @mrousavy I dont think its an issue with the paint or shader, as
- the Example App uses the inverted colour shader from the docs
- These shaders work well on newer hardware (Samsung S23 Ultra)
Granted, the device I'm testing on is very old (Huawei P10 was released 2017). I was wondering if it could be due to these 2 runtime errors as seen in the logs, and would there be any ways to catch them.
2024-06-04 16:49:17.184 8524-8524 GLConsumer com.mrousavy.camera.example E [SurfaceTexture-0-8524-1] checkAndUpdateEglState: invalid current EGLContext
2024-06-04 16:49:17.185 8524-8524 surfaceTexture com.mrousavy.camera.example E Exception updateTexImage java.lang.IllegalStateException: Unable to update texture contents (see logcat for details)
@wcandillon What are some information specifically that i could provide about the phone? Also, how would i be able to check the performance of makeImageFromPlatformBuffer?
Ah interesting - i didnt read those logs my bad. It looks like an error, yep. Not sure if it's RN Skia related or VisionCamera related tho
@wcandillon there's quite a few places in the codebase where RN Skia just swallows errors and logs them to the console (and returns null, or doesnt do anything) - I think throwing an Error (which will be propagated to JS) is better, especially in this case here.
I too get a blank screen, maybe different from OP...when I start my Android device - after running a full build on EAS - I get an error on my device:
Invariant Violation: TurboModuleRegistry.getEnforcing(...): 'RNSkiaModule' could not be found. Verify that a module by this name is registered in the native binary.Bridgeless mode: false. TurboModule interop: false. Modules loaded: {"NativeModules":["PlatformConstants","LogBox","SourceCode","Timing","AppState","BlobModule","WebSocketModule","DevSettings","DevToolsSettingsManager","Networking","Appearance","DevLoadingView","HeadlessJsTaskSupport","DeviceInfo","UIManager","DeviceEventManager","RNCSafeAreaContext","IntentAndroid","NativeAnimatedModule","SoundManager","I18nManager","ImageLoader","RNCAsyncStorage","StatusBarManager","StatusBarManager"],"TurboModules":[],"NotFound":["NativePerformanceCxx","NativePerformanceObserverCxx","RedBox","BugReporting","LinkingManager","PlatformLocalStorage","RNC_AsyncSQLiteDBStorage","FrameRateLogger","KeyboardObserver","ModalManager","Worklets","RNSkiaModule"]}, js engine: hermes
I thought to try my emulator and it complains about minSdkVersion on Android needing to be on 26.
All I get is a white screen for my camera view.
"react-native": "0.73.6",
"react-native-vision-camera": "^4.0.1",
"react-native-worklets-core": "^1.3.0",
"react-native-fast-tflite": "^1.2.0",
"vision-camera-resize-plugin": "^3.1.0"
I updated all the packages related to camera/frameprocessor/worklets, and now camera is working.
EDIT: Actually, I am unable to use useSkiaFrameProcessor on my Android device.
ERROR [session/recoverable-error: An unknown error occurred while creating the Camera Session, but the Camera can recover from it.]
I'm having a somewhat similar issue on OnePlus5 on 4.0.1. In my case the frame is partially displayed
The camera preview is displayed correctly without any issues if I'm using useFrameProcessor:
but once I switch to useSkiaFrameProcessor I get this: (notice the white space below and the L shaped black space
Hi @meleffendi you are lucky at least getting something to viewed. I am getting blank screen. @mrousavy @wcandillon wcandillon guys anyone can help me to fix this issue?
Did any of you guys manage to display anything with your camera when using useSkiaFrameProcessor?
Hey thanks for replying! I did not forget, I am actually using exactly the code you provided in the guide as an example:
const invertColorsFilter = Skia.RuntimeEffect.Make(`
uniform shader image;
half4 main(vec2 pos) {
vec4 color = image.eval(pos);
return vec4((1.0 - color).rgb, 1.0);
}
`)
const shaderBuilder = Skia.RuntimeShaderBuilder(invertColorsFilter)
const imageFilter = Skia.ImageFilter.MakeRuntimeShader(shaderBuilder, null, null)
const paint = Skia.Paint()
paint.setImageFilter(imageFilter)
const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
frame.render(paint)
}, [paint])
doesn't work whether I use front or back camera on emulator; then on real device it does work, but for back camera only, not the front one. Also, regardless of what's displayed, when I record and play the video it's like no filter was applied at all. (separate issue obviously, but still) I see on your documentation that you only do this on a paid basis? (fair enough, but just wanted to make sure it isn't meant to be part of the default package? https://react-native-vision-camera.com/docs/guides/skia-frame-processors)
Try this. This works on 4.5.2. Haven't tested on other versions.
diff --git a/node_modules/react-native-vision-camera/src/skia/useSkiaFrameProcessor.ts b/node_modules/react-native-vision-camera/src/skia/useSkiaFrameProcessor.ts
index 8c8a92d..79c3e8c 100644
--- a/node_modules/react-native-vision-camera/src/skia/useSkiaFrameProcessor.ts
+++ b/node_modules/react-native-vision-camera/src/skia/useSkiaFrameProcessor.ts
@@ -9,6 +9,7 @@ import { SkiaProxy } from '../dependencies/SkiaProxy'
import { withFrameRefCounting } from '../frame-processors/withFrameRefCounting'
import { VisionCameraProxy } from '../frame-processors/VisionCameraProxy'
import type { Orientation } from '../types/Orientation'
+import { Platform } from 'react-native'
/**
* Represents a Camera Frame that can be directly drawn to using Skia.
@@ -89,18 +90,19 @@ function withRotatedFrame(frame: Frame, canvas: SkCanvas, previewOrientation: Or
try {
// 2. properly rotate canvas so Frame is rendered up-right.
const orientation = relativeTo(frame.orientation, previewOrientation)
+ if(Platform.OS === 'ios'){
switch (orientation) {
case 'portrait':
- // do nothing
+ canvas.translate(0, frame.height / 2)
break
case 'landscape-left':
// rotate two flips on (0,0) origin and move X + Y into view again
- canvas.translate(frame.height, frame.width)
+ canvas.translate(0, frame.width)
canvas.rotate(270, 0, 0)
break
case 'portrait-upside-down':
// rotate three flips on (0,0) origin and move Y into view again
- canvas.translate(frame.width, frame.height)
+ canvas.translate(frame.width, frame.height * 1.5)
canvas.rotate(180, 0, 0)
break
case 'landscape-right':
@@ -111,6 +113,33 @@ function withRotatedFrame(frame: Frame, canvas: SkCanvas, previewOrientation: Or
default:
throw new Error(`Invalid frame.orientation: ${frame.orientation}!`)
}
+ } else {
+ switch (orientation) {
+ case 'portrait':
+ canvas.rotate(-90,0,0)
+ canvas.scale(1,-1)
+ canvas.translate(-frame.width, -frame.height)
+ break
+ case 'landscape-left':
+ // rotate two flips on (0,0) origin and move X + Y into view again
+ canvas.rotate(-90,0,0)
+ canvas.scale(1,-1)
+ canvas.translate(-frame.width, -frame.height)
+ break
+ case 'portrait-upside-down':
+ // rotate three flips on (0,0) origin and move Y into view again
+ canvas.translate(frame.width, frame.height * 1.5)
+ canvas.rotate(180, 0, 0)
+ break
+ case 'landscape-right':
+ // rotate one flip on (0,0) origin and move X into view again
+ canvas.translate(frame.height, 0)
+ canvas.rotate(90, 0, 0)
+ break
+ default:
+ throw new Error(`Invalid frame.orientation: ${frame.orientation}!`)
+ }
+ }
// 3. call actual processing code
func()
In my case I had the frame mirrored horizontally on Android in addition to the offset. iOS on the other hand had no issues.
Thanks for sharing your suggestion @meleffendi unfortunately it didnt change anything; still a blank screen on both front and back camera on emulator; and only back camera working on device