Not showing anything if used with react-native-skia

[nmongiya]

I am trying to use react-native-skia with @babylonjs/react-native(1.6.1) and RN 0.71 but can not use both of them together. Only of them is visible on the screen.

I had the same problem earlier with RN 0.70 where skia drawings were not available in debug mode(or when app was connected to metro) but ONLY with reract-native-skia version 0.1.157, I have managed to have both libraries available at the same time.

It prints out the below message but doesn't render anything in EngineView BJS -: Babylon Native (v5.42.2) launched I can't think of a reason why this is happening, any thoughts? is it the canvas that is blocked?

[carolhmj]

ping @CedricGuillemet

[CedricGuillemet]

I don't know anything about react-native-skia. TBH, I didn't even know it existed. It's hard to have a clue on what to check when you don't know how it's architectured. Do you know anything about that @bghgary @ryantrem ?

[ryantrem]

I also don’t really know anything about it. I expect it would require some debugging. @nmongiya are you testing on Android or iOS? Depending on the platform someone might be able to give some guidance on how to try to diagnose. There might be some debugging tips you can reuse here: https://forum.babylonjs.com/t/im-trying-to-make-babylonreactnative-to-work-on-macos/39178/74

[nmongiya]

I am having this issue on Android only. NO issues on ios. Couple of interesting things:

1. When Metro is connected : Skia component is available but Babylon is not
2. When Metro is NOT connected: Babylon is visible on screen but skia is not

I thought it could be a race condition to get the canvas and I tried to load Babylon component after couple of seconds timeout and both works fine in that case even when metro is connected, here is what I can see in android logcat

I/SurfaceView: onWindowVisibilityChanged(0) true android.view.SurfaceView{f45f322 V.E...... ......I. 0,0-0,0} of ViewRootImpl@9e07a49[MainActivity] I/SurfaceView: onWindowVisibilityChanged(0) false android.view.SurfaceView{f2911b3 I.E...... ......I. 0,0-0,0} of ViewRootImpl@9e07a49[MainActivity] I/SurfaceControl: nativeRelease nativeObject s[526794639040] I/SurfaceControl: nativeRelease nativeObject e[526794639040] I/SurfaceControl: assignNativeObject: nativeObject = 0 Surface(name=null)/@0x8c0dc8b / android.view.SurfaceControl.readFromParcel:1117 android.view.IWindowSession$Stub$Proxy.relayout:1820 android.view.ViewRootImpl.relayoutWindow:9005 android.view.ViewRootImpl.performTraversals:3360 android.view.ViewRootImpl.doTraversal:2618 android.view.ViewRootImpl$TraversalRunnable.run:9971 android.view.Choreographer$CallbackRecord.run:1010 android.view.Choreographer.doCallbacks:809 android.view.Choreographer.doFrame:744 android.view.Choreographer$FrameDisplayEventReceiver.run:995 I/SurfaceControl: nativeRelease nativeObject s[526792490368] I/SurfaceControl: nativeRelease nativeObject e[526792490368] I/SurfaceControl: nativeRelease nativeObject s[526793921536] I/SurfaceControl: nativeRelease nativeObject e[526793921536] I/ViewRootImpl@9e07a49[MainActivity]: Relayout returned: old=(0,0,1920,1200) new=(0,0,1920,1200) req=(1920,1200)0 dur=11 res=0x1 s={true 529446535168} ch=false fn=194 I/SurfaceView: surfaceCreated 1 #8 android.view.SurfaceView{f45f322 V.E...... ......ID 0,0-1917,1002} I/SurfaceView: surfaceChanged (1917,1002) 1 #8 android.view.SurfaceView{f45f322 V.E...... ......ID 0,0-1917,1002} I/SurfaceView: applySurfaceTransforms: t = android.view.SurfaceControl$Transaction@b534370 surfaceControl = Surface(name=SurfaceView - com.emesent/com.emesent.MainActivity@f45f322@0)/@0x9a006e9 frame = 194 I/ReactNativeJS: BJS - [13:49:01]: Babylon Native (v5.42.2) launched

[CedricGuillemet]

when both work fine, do you mean both can draw to the canvas? when Skia is available but babylon is not, does it mean babylon still renders but result is not visible or does it mean the render loop is not started?

[nmongiya]

1. Yes, When I said "Both work fine", I meant that both can draw to the canvas.
2. When Skia is available but babylon is not, It meant, babylon still renders but result is not visible(It's just a black screen)
3. Another interesting find is : When I change the androidView='TextureView' in EngineView, both Skia and Babylon works fine in debug mode (even When Metro is connected) but if SKIA gets drawing first, you can not see anything in babylon. :(

[wcandillon]

@CedricGuillemet we also didn't know you were doing such cool things as well :) Let's discuss this offline.

[CedricGuillemet]

3. Another interesting find is : When I change the androidView='TextureView' in EngineView, both Skia and Babylon works fine in debug mode (even When Metro is connected) but if SKIA gets drawing first, you can not see anything in babylon. :(

This is interesting. It looks like a view stacking problem. I bet that running on a different phone will give a different ordering problem. My guess is some views are opaque, some are not. depending on the order, you see something or not. It's possible for the Babylon View to be transparent depending on alpha value. is there the same mecanism with Skia? can you try to force transparency on both views?

[nmongiya]

I tried setting isTransparent={ false } in EngineView but that doesn't help.

[CedricGuillemet]

I would try to have everything transparent: Skia and Babylon

[nmongiya]

Setting both transparent doesn't help either.

I do not how babylon works under the hood for react-native, does it use Canvas as well like Skia and can it be locking issue as they mentioned here https://github.com/Shopify/react-native-skia/blob/main/docs/docs/canvas/contexts.md

[CedricGuillemet]

On Android, depending on transparency, a SurfaceView or a TextureView is created: https://github.com/BabylonJS/BabylonReactNative/blob/aafbad864d6eb6b08c4461caa8d77583e9aa4da3/Modules/%40babylonjs/react-native-iosandroid/android/src/main/java/com/babylonreactnative/EngineView.java#L99

Same mecanism as iOS where a MTKView inheritance is created and transparency is enabled by a flag.

I guess Skia creates a view as well. I believe it's possible to get view hierarchy with AndroidStudio and compare when rendering correctly or not.

[nmongiya]

View hierarchy with AndroidStudio's Layout inspector is not very helpful. It doesn't show view attributes that we are looking for.

[chrfalch]

Hi! RNSkia always uses a TextureView when rendering - could the issue here be that combining TextureViews (Skia) and SurfaceViews (Babylon) causes this behaviour?

Did anyone see a difference when setting transparency on the Babylon View to force it to render a TextureView?

[CedricGuillemet]

TextureView allows transparency. SurfaceView is opaque. That's the reason why setting transparency flag on the engineView creates a textureView. So to display BRN on top of Skia, BRN View must be set as transparent.

[nmongiya]

Hi @CedricGuillemet I have tried with the following code, with or without setting the androidView. As soon as there is an update in Skia component, BRN component freezes. <EngineView camera={camera} onInitialized={onInitialized} displayFrameRate={true} isTransparent={true} androidView="TextureView" antiAliasing={2} />

Please find attached a screen recording illustrating how Skia and Babylon Native are fighting for (canvas?) resources. The foreground red square in the top left corner is the Skia component and the Babylon 3D scene is in the background. As you'll see in the video, when I don't interact with the Skia component, Babylon just works fine - I can rotate around the scene in Babylon and start the XR (camera video stream). However as soon as I tap the Skia component, the Babylon component freezes. I need to toggle the Babylon window to restore it. Another interesting thing is, when the Skia component takes control and I attempt to start the XR, it opens the camera video stream in the Skia component (instead of Babylon) until I reset it back by toggling the Babylon view.

https://github.com/BabylonJS/BabylonReactNative/assets/24381573/3bca6f52-dd9e-4f34-a706-eed071d03244

[CedricGuillemet]

Looks like Skia like to work alone :) I'm wondering if there is some option to make it less exclusive

[wcandillon]

@CedricGuillemet I sent you some messages privately in case you want to connect about this topic.

[CedricGuillemet]

@CedricGuillemet I sent you some messages privately in case you want to connect about this topic.

I've been busy this week. I'll contact you next one.

[chrfalch]

We've closed this in the RN Skia repo since RN Skia renders nicely with modules like Expo-gl (reproduction can be found here: https://github.com/chrfalch/ExpoGLAndSkia).

As the documentation states, Babylon.js only works a single view at a time - guess this is also the limitation you meet when trying to combine it with RN Skia.

Our issue is now closed with the above info.

[bghgary]

Closing for now. Please reopen if this is still needed.