MixedReality-WebRTC
MixedReality-WebRTC copied to clipboard
microsoft
Can I get two source from WebcamSource?
Hi, I am using this plugin to stream mixed reality capture from HoloLens to a desktop application. I am wondering if there is a way to obtain two source (cam with hologram and without hologram ) at the same time, because I want to use the clean image to do AI detection. I know it is possible to stream cam & hologram (SceneVideoSource) seperatly and map together, but we want to avoid the mapping and offset problem at first place. Also I noticed that DevicePortal can stream the cam without hologram when the app is streaming with webRTC, so maybe it is possible to obtain two source simultaneously? appreciate your help!
I'm open to correction on this, but to my knowledge, I don't think this is possible (at least on HoloLens)
The Mixed Reality Capture is implemented as a 'Video Effect' on the webcam stream, and to the best of my knowledge, you cannot read the same video stream with different settings in UWP. While it is possible to have multiple streams from the same camera, they must all use the same settings (e.g. they must be in Shared mode, not Exclusive).
If you had a second camera, it is certainly possibly to read both streams at the same time, but I don't think that is practical when working with HoloLens.
Also I noticed that DevicePortal can stream the cam without hologram when the app is streaming with webRTC, so maybe it is possible to obtain two source simultaneously?
That is certainly interesting, maybe it is possible then? It could just be that the Device Portal has special behaviour/access.
The short answer is I don't know, beyond what @spacecheeserocks already described, which is also my understanding. It is unclear to me if you can open the camera twice (in Shared mode) and have the MRC Video Effect on one only. Chances are that you cannot. I would not look too much at the Device Portal, which has possibly some special access indeed (I don't know really, but wouldn't be surprised).
@spacecheeserocks @djee-ms Thanks both for your reply. I will give a try to use additional MediaCapture (in Shared mode) to capture images without MRC effect.
While it is possible to have multiple streams from the same camera, they must all use the same settings (e.g. they must be in Shared mode, not Exclusive).
By the way, I found that Unity's PhotCapture can obtain images without MRC when WebRTC running, but it will cause significant lag and jitter on HoloLens. Unfortunately It does not meet my expectations.
Thanks again, I will keep trying.
not sure if it is related:
i'm trying to get multiple webrtc sessions to run, and currently the unity editor crashes as soon as i have more than 1 webcamsource active in the scene (basically the videochatemo and duplicating the remote).
use additional MediaCapture (in Shared mode) to capture images without MRC effect.
It works! But, I am still struggle with lag and jitter on HoloLens(fps is down to 20~25), because I try to encode softwareBitmap to jpg byte array. I am going to find out how to get image byte from Direct3DSurface to improve fps.
use additional MediaCapture (in Shared mode) to capture images without MRC effect.
It works! But, I am still struggle with lag and jitter on HoloLens(fps is down to 20~25), because I try to encode softwareBitmap to jpg byte array. I am going to find out how to get image byte from Direct3DSurface to improve fps.
Hello, have you successfully used Direct3DSurface to obtain image byte arrays? I am also using BitmapEnconder to convert the softwarebitmap to JPEG byte array, but the fps is down to 15 - 20. If you are successful, can you please introduce it in detail, thank you.
@zxzkf1992 Unfortunately, I have not succeeded yet. If there is any progress in the future, I will post it here.
@zxzkf1992 i've moved to the unity webrtc implemtation. though it's still in early preview it worked much better (audio is not supported yet though) and works with 2 webcams as well.