Webgl not working?[BUG]

[MarcoLavoro]

Describe the bug I followed this guide: https://github.com/Unity-Technologies/com.unity.webrtc/blob/develop/Documentation~/install.md

downloaded the example and try to build with webgl the scene: MultiplePeerConnections But i get over 200 errors like this: Assets\Samples\WebRTC\2.3.3-preview\Example\MungeSDP\MungeSDPSample.cs(24,13): error CS0246: The type or namespace name 'RTCOfferOptions' could not be found (are you missing a using directive or an assembly reference?)

To Reproduce Install unity 2019.4.11 follow the guide https://github.com/Unity-Technologies/com.unity.webrtc/blob/develop/Documentation~/install.md downloaded the example and try to build with webgl the scene

Expected behavior I can compile

Environment (please complete the following information):

• OS: Windows
• Unity Version [e.g. 2019.4.11f]

Is normal or I am doing something wrong?

[gtk2k]

WebGL is not (yet?) supported. https://github.com/Unity-Technologies/com.unity.webrtc#roadmap

[karasusan]

Yeah, WebGL is not supported.

[marcin-aicradle]

What would it take to add WebGL support? Is there a technology restriction or it simply requires to add some code? I may be potentially willing to add and use it.

[gtk2k]

@marcin-aicradle I've tried porting com.unity.webrtc to WegGL before myself, but the texture rendering of VideoStreamTrack is very heavy. I think the port to WebGL is probably tough. If you have any knowledge that you can render VideoStreamTrack (MediaStreamTrack) at high speed, please let me know.

[marcin-aicradle]

Thanks for the answer. Unfortunately I don't have such knowledge.

[antoined73]

@marcin-aicradle I've tried porting com.unity.webrtc to WegGL before myself, but the texture rendering of VideoStreamTrack is very heavy. I think the port to WebGL is probably tough. If you have any knowledge that you can render VideoStreamTrack (MediaStreamTrack) at high speed, please let me know.

I'm currently trying to do so as well since we need video streaming support via WebRTC in WebGL builds. Do you have any examples on how you made it please ?

It's not really clear how I will do that but I'm thinking about :

• WebRTC connection is done in a jslib
• The jslib gets the video stream
• The jslib sends a byte array to a c# bridge callback in unity
• And then from that byte array I should be able to render it has a render texture ?

I don't know if it will need some decoding. And if so, how to do that. Maybe using the decoders from the com.unity.webrtc package.

Thanks

[gtk2k]

@antoined73 Yes, I also implement it that way. Here is the source that I made WebGL compatible, so please take a look. https://github.com/gtk2k/com.unity.webrtc/tree/feature/WebGL

Note: The original version of com.unity.webrtc that I implemented a long time ago is out of date.

[antoined73]

@gtk2k Thanks for you answer !

I tried to execute your version in my editor, but unfortunately it didn't worked : the videoReceiver scene log an error when clicking the "call" button Uncaught ReferenceError: _CreatePeerConnectionWithConfig is not defined at _ContextCreatePeerConnectionWithConfig.

Also I had to remove the Tests and Editor folder in order to make it compile.

If I understood correctly, the video transfer is done is this method :

UpdateRendererTexture: function (trackPtr, renderTexturePtr, needFlip) {
    // console.log('UpdateRendererTexture');
    // if (!uwcom_existsCheck(trackPtr, 'UpdateRendererTexture', 'track')) return;
    console.log('UpdateRendererTexture', renderTexturePtr);
    var video = uwcom_remoteVideoTracks[trackPtr].video;
    var tex = GL.textures[renderTexturePtr];
    GLctx.bindTexture(GLctx.TEXTURE_2D, tex);
    if (!!needFlip){
      //GLctx.pixelStorei(GLctx.UNPACK_FLIP_Y_WEBGL, true);
    }
    GLctx.texImage2D(GLctx.TEXTURE_2D, 0, GLctx.RGBA, GLctx.RGBA, GLctx.UNSIGNED_BYTE, video);
    // GLctx.texSubImage2D(GLctx.TEXTURE_2D, 0, 0, 0, GLctx.RGBA, GLctx.UNSIGNED_BYTE, video);
    GLctx.texParameteri(GLctx.TEXTURE_2D, GLctx.TEXTURE_MAG_FILTER, GLctx.LINEAR);
    GLctx.texParameteri(GLctx.TEXTURE_2D, GLctx.TEXTURE_MIN_FILTER, GLctx.LINEAR);
    GLctx.texParameteri(GLctx.TEXTURE_2D, GLctx.TEXTURE_WRAP_S, GLctx.CLAMP_TO_EDGE);
    GLctx.texParameteri(GLctx.TEXTURE_2D, GLctx.TEXTURE_WRAP_T, GLctx.CLAMP_TO_EDGE);
    GLctx.bindTexture(GLctx.TEXTURE_2D, null);
  }

I think you may not be very far from a solution that could work with great performances. In fact, I thought at first you made it by passing a byte array from js to c#, but instead your are directly binding the webgl texture to unity texture (if i understood correctly).

For further optimization, maybe we could directly move the image data from an HTML video element to a WebGL texture ? I don't know if it makes sense, but I've seen an WebRTC asset on the store that was talking about that.

[gtk2k]

@antoined73

maybe we could directly move the image data from an HTML video element to a WebGL texture ?

That code is the code that directly transfers the video element.

GLctx.texImage2D(GLctx.TEXTURE_2D, 0, GLctx.RGBA, GLctx.RGBA, GLctx.UNSIGNED_BYTE, video);

The video variable is the HTML video element.

[antoined73]

@gtk2k Oh okay, my bad. I don't have any ideas how to make it work faster then.

What are the actual perfomances you managed to have ? Something like 0-10 or 10-25 fps ?

I'm currently trying to update your code to the latest 2.4.0-exp.1 release. Wish me luck 😅

[gtk2k]

@antoined73 It's 0-10.

[antoined73]

@gtk2k What are the reasons you changer the Lib path for editor windows in WebRTC.cs ? to "Assets/Runtime/Plugins/x86_64/webrtc.dll" instead of "Packages/com.unity.webrtc/Runtime/Plugins/x86_64/webrtc.dll" ?

Edit : Also, I'd like to make your initial feature work, but I can't... Is it normal that the Editor and Tests folders are not compiling when trying to build to webgl ? Also the exemple scenes (dataChannel, bandwitdh, trickleice, stats and changecodecs) does not compile.

I'm trying just to run the VideoReceiver example scene which compiles but show a blocking error I referenced earlier :

the videoReceiver scene log an error when clicking the "call" button Uncaught ReferenceError: _CreatePeerConnectionWithConfig is not defined at _ContextCreatePeerConnectionWithConfig.

If you could give me some help, I would greatly appreciate it ! Thanks 😄

[gtk2k]

@antoined73 The goal is to make the PC code work almost as-is in WebGL builds, so you need to modify the source code, and with Packages / ~, even if you modify the code, it will return to the original. .. If you know any other workarounds, please let me know.

[gtk2k]

@antoined73

I pushed the sample project to GitHub. https://github.com/gtk2k/UnityWebRTC_WebGL_Sample_Project

Try building and running WebGL in the scene called MultiVideoReceive in this project.

[antoined73]

Okay, super great the sample project works ! 🥳

Could you explain to me what these methods are referencing to ? In Context.jslib :

ContextCreateVideoTrack: function (contextPtr, srcTexturePtr, dstTexturePtr, width, height) {
    if (!uwcom_existsCheck(contextPtr, 'ContextCreateVideoTrack', 'context')) return;
    return _CreateVideoTrack(srcTexturePtr, dstTexturePtr, width, height);
  },

I don't understand where is the method "_CreateVideoTrack" which is beeing called. Can't find it anywere. Same for "_PeerConnectionSetDescription", "_CreatePeerConnectionWithConfig", ect. All the methods beginining with " _ " in .jslibs.

[gtk2k]

CreateVideoTrack function in VideoStreamTrack.jslib.

Functions in .jslib are prefixed with "_" when built. (emscripten)

[antoined73]

I'm currently trying to make the webgl webrtc plugin work with a custom signaling server. I'm really close to make it work, but i'm currentrly stuck on this : RTCIceCandidate rtc_ice = new RTCIceCandidate(new IntPtr(), candidate, null, sdpMLineIndex);

Since I use a custom signaling server, I receive the sdpMLineIndex and candidate via websockets and create a RTCIceCandidate from that. In the official webRTC lib, you can create a RTCIceCandidate via a RTCIceCandidateInit object (which contains sdpMLineIndex and candidate). In the webGL version, I see you changed the constructor of this object to accept a pointer instead. Which pointer do I need to put there ? By giving a new IntPtr the render loop runs, but of course nothing displays and an error occurs because webRTC can't find the ice candidate.

If you have any ideas, I'll be glad to hear from you, Thanks you !

[devluz]

@gtk2k I haven't looked through your code yet but you can usually reach a good speed by moving the image from a VideoTag directly into a Texture: https://github.com/because-why-not/awrtc_browser/blob/master/src/awrtc/media/RawFrame.ts#L193

[gtk2k]

The poor performance is not on the receiving side, but rather on the sending side, where the biggest problem is getting pixel data from RenderTexture and streaming it.

[gtk2k]

@antoined73 currently creating a signaling sample including source code modifications, so please be patient.

[devluz]

@gtk2k do you plan to stream a specific texture or the unity screen? If you stream the complete canvas it can be quite fast: https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/captureStream It might be possible you find a way to use this for individual cameras as well but you would somehow render it to the Canvas first.

[gtk2k]

@antoined73 It's still not working perfectly and I get an error, but I've managed to send a VideoStream using signaling for the time being. Unity-> Web can send the video, but for some reason Web-> Unity raises the OnTrack event but the video is not displayed.

https://github.com/gtk2k/UnityWebRTC_WebGL_Sample_Project

1. In this repository, there is a folder called UseSignalingSample in the Assets-> Samples folder, open the UseSamples scene in the Scenes folder in it, build WebGL and execute it.

2. There is a folder called SampleSignalingServer directly under the project folder, and the code of the sample signaling server created by Node.js is there. Run to start the signaling server.

3. The signaling server is also a web server at the same time, so if you access http: // localhost: 8989, the web version of the client will be displayed.

4. Click the Connect button on each of the Web version and Unity WebGL version to start the WebRTC connection and display the video.

[gtk2k]

@devluz It is heavy to copy the RenderTexture image to the canvas, that is, to get the RenderTexture pixel data every frame.

[antoined73]

@gtk2k Thank you very much for your help ! I kept my custom signaling server and took the bits I missed in the API (CreateNativeRTCIceCandidate) and it works super smoothly ! For your information, I use Unity WEBGL to receive a video from a custom server using Gstreamer and it works great.

I think I will use this sample project, upgrade it with the latest 2.4.0-exp.1 version that supports Android, and I will be able to consume a video stream via webRTC in every plateforms 🥳 ! (Android, Desktops, WebGL)

[fenos]

@antoined73 @gtk2k Thanks a lot for your contributions! Would it be possible for you to contribute back with a PR to this project and provide support for WebGL out of the box?

[gtk2k]

@fenos Thanx.

The current implementation of the WebGL version does not fully implement the functionality of Unity.WebRTC. The first goal was to create a minimal implementation that would work with very little modification to the Unity.WebRTC sample code. The current WebGL version is based on the older Unity.WebRTC version. For example, the new Unity.WebRTC deprecates the createOffer and createAnswer method arguments (offerOptions/answerOptions), but the WeBGL version can still include them. (It's a WebGL version, so it doesn't really matter ...) Therefore, the new Unity.WebRTC is basically supposed to use a walkie-talkie. The WebGL version does not properly implement transceiver related. Also, the implementation of statistics (Stats) is almost unimplemented. Unfortunately, it is difficult to issue a PR immediately because it is necessary to implement unimplemented functions before issuing a PR.

[antoined73]

@fenos gtk2k is really the only one to have worked on that subject. I'm only using what he has done so far. I'll be glad to help if I can, since I need this WebGL support on the latest version of the official lib for work purposes. But I'm afraid I don't have enought knowledge on how the lib works to do it all by myself.

@gtk2k, if you want to collaborate on this it could be great.

We could work on that and issue a PR when we are done. The missings bits not really vital (like stats) can be done later. Finally, I could try but I'm not sure if will have the time and knowledge to provide any support either.

[antoined73]

Update : currently working on it there : https://github.com/antoined73/com.unity.webrtc/tree/feature/webgl-support I merged the 2 versions (official unity's webrtc lib + @gtk2k version for WebGL support).

The library is compiling correctly in windows editor no matter which build plateform you target. My goal for this week is to make a successfull build for all the supported platforms with WebRTC. Then, I'll see if WebGL need some more work to make it isofunctionnal with the latest official lib version.

If you want to provide any help, you are most than welcome. Cheers !

[nvanofwegen]

@antoined73 I'm also working on converting gtk2k repo to the current webrtc lib. Mostly focusing on the RTCDataChannels. Currently there is some issue with sending data between browsers, but browser -> editor/standalone and editor/standalone -> browser do work. I'm also fairly new to WebRTC so I'm still a bit fighting with the whole negotiation part, so maybe that's the reason it doesn't work between browsers yet. Since it still doesn't work completely I'm unsure about a PR, but it's also a bit useless to make some jslib changes multiple times individually.

[antoined73]

Oh, okay ! I think you must be further in the development than I, because I am not having a successful WebGL build yet. I'm new to WebRTC too, so that's quite a challenge indeed. I don't intend to use WebRTC between unity builds, but rather using a custom signaling server to connect a unity build to a linux c++ media streamer server (managed to do that with the latest sample gtk2k provided).

Do you mind sharing your repository ? Maybe I could help you from there, if I can do it.

For my part : I'm still ending the merge of the 2 versions. I stubbed the stats and tests part for webgl and I stubbed some API on WebRTC.cs that are requested by the Example scenes. Several API from the WebRTC.cs have been removed/changed when compiling to WebGL, but I think its better if we can keep the same API in order to make it totally transparent for the final user. So for now, I kept most of the gtk2k modifications with some stubs here and there to make a successfull compilation. But I definitely think there will be a need to change the .jslib then.

PS : I think you could make a PR anyway, people will be aware that work is done toward this new build plateform, and therefore we could enhance this support together. Even if its not accepted right away, it still shows to unity that we're working on that.