[BUG]: Not able to display MediaStream from browser inside unity on RawImage as texture.

[rmoback]

Package version

2.4.0-exp.6

Environment

* OS:Window10
* Unity version: 2019.4.17f1
* WebRTC Package : 2.4.0-exp.6

Steps To Reproduce

Grab a media stream from browers WebRTC api's and add that stream to remotepeerconnection. Following is the JavaScript code from browser :

const localConnection = new RTCPeerConnection()
 
localConnection.onicecandidate = e =>  {
console.log(" NEW ice candidnat!! on localconnection reprinting SDP " )
 console.log(JSON.stringify(localConnection.localDescription))
}

const sendChannel = localConnection.createDataChannel("sendChannel");
 sendChannel.onmessage =e =>  console.log("messsage received!!!"  + e.data )
   sendChannel.onopen = e => console.log("open!!!!");
     sendChannel.onclose =e => console.log("closed!!!!!!");

handleSuccess = stream => {
            for (const track of stream.getTracks()) {
                localConnection.addTrack(track, stream);
            }
        }
        handleError = e => {
            console.log("error : ",e)
        }
await navigator.mediaDevices.getDisplayMedia({ video: true, audio: false }).then(handleSuccess, handleError);
localConnection.createOffer().then(o => localConnection.setLocalDescription(o) )

Next add the offer generated by the browser into unity and create answer, set the answer in browser and connection will be established. Once the connection is established I can see the streams are activate in chrome://webrtc-internals/ and frames are being transmitted. to remotepeer.

Unity Side once you receive the video add the texture to rawImage : The result is grey color image has been displayed not the correct one. All other parameters are correct as I tested height and width of texture being received in unity.

Here texture assign is the code from unity:

private DelegateOnTrack remoteOntrack;

remoteOntrack = e =>
             {  
                 if (e.Track is VideoStreamTrack video)
                 {
                     video.OnVideoReceived += tex =>
                     {
                         receiveImage.texture = tex;
                     };
                     receiveVideoStream = e.Streams.First();
                     receiveVideoStream.OnRemoveTrack = ev =>
                     {
                         receiveImage.texture = null;
                         ev.Track.Dispose();
                     };
                 }
             };

remotePeerConnection.OnTrack = remoteOntrack;

Current Behavior

Browser sends the displayMedia as a stream to remotepeer to unity and Texture rendered on RawImage inside unity is grey color image.

Expected Behavior

Correct displayMedia tobe displayed on the rawImage of unity scene.

Anything else?

No response

[karasusan]

@rmoback We have never tested the combination getDisplayMedia and Unity. We will add the test case.

[karasusan]

memo URS-427

[rmoback]

Also I checked with browser's navigator.mediaDevices.getUserMedia which streams camera feed is also not working.

[karasusan]

@rmoback Does the other sample work correctly?

[rmoback]

Only I can able to stream data channel from browser webRTC to unity webRTC, none of the video streams works.

[kannan-xiao4]

@rmoback Maybe it's a RenderStreaming issue, can you tell me the version of com.unity.renderstreaming?

Also, please try running the Bi-directional sample with Renderstreaming. https://docs.unity3d.com/Packages/[email protected]/manual/sample-bidirectional.html

This sample has been confirmed to work, so if you can't receive the video even with this sample, please attach Editor.log or the log of the console in the browser.

[rmoback]

@kannan-xiao4 I am not using unity's RenderStreaming library. I am trying to accomplish a simple video stream with Unity's webRTC package and browser's native javascript WebRTC protocols.

[kannan-xiao4]

@rmoback I see, But I don't know your implementation, can you try the Render Streaming sample first? (Bidirectional sample) Receiving the getusermedia stream in browser should be fine.

[kannan-xiao4]

@rmoback Do you try bidirectional sample ? (with Unity and Browser combination)

[gtk2k]

Is Chrome on the browser side? In case of Chrome, if the browser side is not accessing by HTTPS, getUserMedia() will not work unless you click the icon on the left of the address bar and click the reset authority button to reload.

[rmoback]

@kannan-xiao4 I tried bidirectional sample its working from browser to Unity. But what I am trying here is using WebRTC package not using unity render streaming. @gtk2k I didnt get you what reset authority button means, I didnt see any buttons like that, can you please share and screen shots ?

I am attaching a simple video for what I am trying, please let me know if I am going somewhere wrong.

https://user-images.githubusercontent.com/77753827/165407534-04b5456b-55ba-48b9-81de-590f0ad86f17.mp4

In video I am trying to initiate the webRTC peer from browser in console, once the screen started sharing you will receive a offer generated, copy that offer and paste it in unity under ExternalVideo script attached to GameObject. Run the unity and click Set Offer button, the white sqaure above the unity is a RawImage. once you click on Set offer button it generated answer in console copy that answer and paste it in Browser. the connection will establish between unity and browser and the image turns grey rather than turning the stream image from browser.

Browser Code :

Step : 1 :

const localConnection = new RTCPeerConnection()
 

localConnection.onicecandidate = e =>  {
console.log(" NEW ice candidnat!! on localconnection reprinting SDP " )
 console.log(JSON.stringify(localConnection.localDescription))
}


const sendChannel = localConnection.createDataChannel("sendChannel");
 sendChannel.onmessage =e =>  console.log("messsage received!!!"  + e.data )
   sendChannel.onopen = e => console.log("open!!!!");
     sendChannel.onclose =e => console.log("closed!!!!!!");

handleSuccess = stream => {
            for (const track of stream.getTracks()) {
                console.log(track);
                localConnection.addTrack(track, stream);
            }
        }

        handleError = e => {
            console.log("error : ",e)
        }

await navigator.mediaDevices.getDisplayMedia({ video: true, audio: false }).then(handleSuccess, handleError);



localConnection.createOffer().then(o => localConnection.setLocalDescription(o) )

2 : Copy paste the offer from browser to unity , press Set Offer button. 3. Copy paste the answer from Unity to browser

const answer = answer from unity

4 : Set the remotedescription in browser with above answer

localConnection.setRemoteDescription (answer).then(a=>console.log("done"))

this completes the connection between Unity and Browser ( also you can test replacing getDisplayMedia with getUserMedia in browser snippet)

ExternalVideo.cs

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Unity.WebRTC;
using UnityEngine.UI;
using System.Linq;
using System;

public class ExternalVideo : MonoBehaviour
{

    [SerializeField] private Button setOfferButton;

    private RTCPeerConnection remotePeerConnection;
    private DelegateOnIceConnectionChange remoteIceConnectionChange;
    private DelegateOnIceCandidate remoteOnIceCandidate;
    private MediaStream receiveVideoStream;
    [SerializeField] private RawImage receiveImage;

    private DelegateOnTrack remoteOntrack;
    public string offer;
    private void Awake()
    {
        WebRTC.Initialize();
        setOfferButton.onClick.AddListener(setOffer);
    }



    private void Start()
    {
        remoteIceConnectionChange = state => { OnIceConnectionChange(remotePeerConnection, state); };
        remoteOnIceCandidate = candidate => { OnIceCandidate(remotePeerConnection, candidate); };
        remoteOntrack = e =>
             {  
                 Debug.Log(e.Track.Kind +"___"+e.Track.Id);
                 if (e.Track is VideoStreamTrack video)
                 {
                    //   receiveImage.texture = video.Texture;
                     video.OnVideoReceived += tex =>
                     {
                         Debug.Log("Texture : W "+tex.width+" H : "+tex.height);
                         receiveImage.texture = tex;
                         Debug.Log("==================================================="+video.Id);
                     };
                     receiveVideoStream = e.Streams.First();
                     receiveVideoStream.OnRemoveTrack = ev =>
                     {
                         receiveImage.texture = null;
                         ev.Track.Dispose();
                     };
                 }
             };
             call();
    }

    private static RTCConfiguration GetSelectedSdpSemantics()
    {
        RTCConfiguration config = default;
        config.iceServers = new[] { new RTCIceServer { urls = new[] { "stun:stun.l.google.com:19302" } } };

        return config;
    }
    private void call()
    {
        var configuration = GetSelectedSdpSemantics();
        remotePeerConnection = new RTCPeerConnection(ref configuration);
        Debug.Log("RemotePeer Created");
        remotePeerConnection.OnIceCandidate = remoteOnIceCandidate;
        remotePeerConnection.OnIceConnectionChange = remoteIceConnectionChange;
        remotePeerConnection.OnTrack = remoteOntrack;

    }
    public struct customRTCSessionDescription
    {
        public string type;
        public string sdp;
    }

    public void setOffer()
    {

        RTCSessionDescription offerSDP = new RTCSessionDescription();
        var offer_SDP = JsonUtility.FromJson<customRTCSessionDescription>(offer);
        Debug.Log(" ----" + JsonUtility.ToJson(offer_SDP));
        if (offer_SDP.type == "offer")
        {
            offerSDP.type = RTCSdpType.Offer;
            offerSDP.sdp = offer_SDP.sdp;
            StartCoroutine(OnCreateOfferSuccess(remotePeerConnection, offerSDP));
        }

    }
    private void OnSetRemoteSuccess(RTCPeerConnection pc)
    {
        Debug.Log($" SetRemoteDescription complete");
    }

    private void OnSetLocalSuccess(RTCPeerConnection pc)
    {
        Debug.Log($" SetLocalDescription complete");
    }

    static void OnSetSessionDescriptionError(ref RTCError error)
    {
        Debug.LogError($"Error Detail Type: {error.message}");
    }
    private static void OnCreateSessionDescriptionError(RTCError error)
    {
        Debug.LogError($"Error Detail Type: {error.message}");
    }
    private IEnumerator OnCreateOfferSuccess(RTCPeerConnection pc, RTCSessionDescription desc)
    {
        var op2 = pc.SetRemoteDescription(ref desc);
        yield return op2;
        if (!op2.IsError)
        {
            OnSetRemoteSuccess(pc);
        }
        else
        {
            var error = op2.Error;
            OnSetSessionDescriptionError(ref error);
        }

        Debug.Log(" createAnswer start");

        var op3 = pc.CreateAnswer();
        yield return op3;
        if (!op3.IsError)
        {
            yield return OnCreateAnswerSuccess(pc, op3.Desc);
        }
        else
        {
            OnCreateSessionDescriptionError(op3.Error);
        }
    }

    IEnumerator OnCreateAnswerSuccess(RTCPeerConnection pc, RTCSessionDescription desc)
    {
        Debug.Log($"Answer: ");
        Debug.Log(JsonUtility.ToJson(desc));
        var op = pc.SetLocalDescription(ref desc);
        yield return op;

        if (!op.IsError)
        {
            OnSetLocalSuccess(pc);
        }
        else
        {
            var error = op.Error;
            OnSetSessionDescriptionError(ref error);
        }
    }

    private void OnIceCandidate(RTCPeerConnection pc, RTCIceCandidate candidate)
    {
        Debug.Log(" ICE candidate "+remotePeerConnection.LocalDescription);
        Debug.Log($"{pc} ICE candidate:\n {candidate.Candidate}");
    }


    private void OnIceConnectionChange(RTCPeerConnection pc, RTCIceConnectionState state)
    {
        switch (state)
        {
            case RTCIceConnectionState.New:
                Debug.Log($"{pc} IceConnectionState: New");
                break;
            case RTCIceConnectionState.Checking:
                Debug.Log($"{pc} IceConnectionState: Checking");
                break;
            case RTCIceConnectionState.Closed:
                Debug.Log($"{pc} IceConnectionState: Closed");
                break;
            case RTCIceConnectionState.Completed:
                Debug.Log($"{pc} IceConnectionState: Completed");
                break;
            case RTCIceConnectionState.Connected:
                Debug.Log($"{pc} IceConnectionState: Connected");
                break;
            case RTCIceConnectionState.Disconnected:
                Debug.Log($"{pc} IceConnectionState: Disconnected");
                break;
            case RTCIceConnectionState.Failed:
                Debug.Log($"{pc} IceConnectionState: Failed");
                break;
            case RTCIceConnectionState.Max:
                Debug.Log($"{pc} IceConnectionState: Max");
                break;
            default:
                throw new ArgumentOutOfRangeException(nameof(state), state, null);
        }
    }

}

[kannan-xiao4]

@rmoback If it works with the RenderStreaming sample, I think it's the Signaling process that has the problem. I can't help because I don't understand how Signaling behavers.

I'd like you to report bugs about only WebRTC Package on GitHub Issue, so please post individual issues in the UnityRenderStreaming forums.

[rmoback]

it is not a signaling process problem as I am able to establish the connection between both the clients , in the video which I posted you can see the track ID been seen in the unity logs once the connection is established. Also I have tested the Data channel transferring the data from between both clients. The only issue with unity WebRTC is video streaming from browser.

As of now I am using UnityRenderStreaming as a workaround, which I assume on the browser side uses peer.js.

[kannan-xiao4]

@rmoback If you can't receive video with getUserMedia in your environment, probably have an implementation problem. (Because it can be received using Render Streaming). Please check RenderStreamingInternal.cs.

If you can't receive the video only with getDisplayMedia, we haven't tested it yet and we'll investigate. https://github.com/Unity-Technologies/com.unity.webrtc/issues/652#issuecomment-1075858066 https://github.com/Unity-Technologies/com.unity.webrtc/issues/652#issuecomment-1075859065

[kannan-xiao4]

@rmoback Hi, Any updates?

[eaphone]

@rmoback seems you are not call WebRTC.update() at receive side. it works fine at my case adding it with your code

[dom98111]

Hi! Have you managed to solve this problem? I get exactly the same problem: a gray texture in the Raw Image, but with the correct height and width. OnVideoReceived is called only 1 time