viromedia
[Feature Req] Render RN View into VR/AR as Texture Source
I can't find any references in the documentation so asking here.
I want to use a webrtc stream as a video texture. Currently only file/url sources are supported it seems.
There's a RN webrtc lib I would like to use https://github.com/oney/react-native-webrtc ) It renders onto its own custom component which looks like it's a glview.
Would it be possible to use that as a texture?
Another way I'm thinking is to save images regularly and trigger the texture update. Would that be possible ? I'm assuming it would not be terribly fast and I'd have to clean up after myself somehow.
Any other ideas ?
Hey @shokimble, thanks for reaching out. Curious to ask - have you tried playing your video source through the ViroVideo component? It uses exoplayer that supports real time video encoding like HLS, but i'm not sure if in the case of webRTC.
Unfortunately we don't yet have a Viro component that renders React Views directly into VR / AR. This solution may work for the use case you've shared above, with some caveats in perf (duplicated rendering) and dimension sizing.
As such there are two solutions:
- Encapsulate the component you wish to render in VR/AR in a "ViroCapture" like view, that in turns renders whatever views are encapsulated onto a texture.
- Another alternative is to modify RN webrtc, such that you can provide it a opengl Texture ID (that we give you, that represents a Viro Texture material), so that RN webrtc can render on. There will be perf savings in this option.
I'll turn this into a Feature request and file a story on our backlog. Are you actively working on shipping an application that will be blocked by this feature?
As it's webrtc you don't get a URL as such so there is nothing to pass into exoplayer. I did a bit of googling to confirm there weren't any examples of passing the buffer directly in somehow but it appears not to be something done before. Makes sense as it has its own player.
Is solution two something achievable today (i.e without modifying Viro React)? It's not something the rtc lib supports but there's a few options around and most are open source so I don't have a problem writing the code for it and submitting a pull request (whether it's accepted is something else).
I'm not actively working on an app but putting together a feasibility study on whether an AR chat app could be done in RN without having to go full Native. It would be great if it was possible.
I'm guessing number 1 is conceptual entirely as I can't find the element you mentioned. There's a few capture component for RN it could be based on but that component can't capture GL views last time I checked which is where cameras and video tend to be rendered.
https://github.com/gre/react-native-view-shot
Ah i see, yes that does make sense.
Yup, both options above are mainly conceptually potential solutions for supporting the rendering of a React View / WebRTC component.
As such, unfortunately neither of them are currently supported by the latest version of the ViroReact platform. Apologies for the inconvenience.
This issue will be updated once a solution has been decided upon, and then again once it has been deployed into release.
No problems. I'll put the idea on the backburner for now. Trying to implement it separately for arcore and arkit would be a lot of work hence why your platform is great 😀
Hello @dthian
I would like to ask if for now WebRTC get supported? Thanks!
Hey @tuananhcwrs, currently Viro React does not support webRTC. Please file a separate github issue / feature request if this is something you would like. Much thanks!
