Webrtc support

[rexroni]

I saw the webrtc branch, but it seems to have not gotten much attention in a long while.

I did some investigating, because I would really like to have calling working. My branch with some debug print statements is here: https://github.com/rexroni/weechat-matrix/tree/webrtc

I was able to determine why the gst webrtc object is failing by building gstreamer from source and running weechat from inside a gstreamer devenv, as described here: https://github.com/GStreamer/gst-build/blob/master/README.md

When I did that, I started to get error messages from gstreamer on stderr. I was able to determine that:

• the set-remote-description event is sent, and in the plugin the _set_description_task is getting queued here
• _set_description_task is not actually getting called immediately
• create-answer event is sent, which results in a call to _create_answer_task, which prints the error message saying "Asked to create an answer without a remote description" here
• Then somehow _set_description_task is called at a later time.

However, I must admit I'm a bit baffled as to just how the gst architecture works. For example: what is the purpose of interrupting promises from emitted events here?

[rexroni]

Ok, a bit more progress, on my webrtc branch. I found the source of the problem with set-remote-description not running; the pipeline had to be started before the set-remote-description event was emitted. I also removed all of the promise.interrupt() calls, I added more debug printing around signals, and I disabled the on-negotiation-needed in when role=="answer", since in all the examples, only the offerer uses it.

It seems that the webrtc object's signaling-state property is correctly being set to STABLE after the local and remote descriptions are set. However, it seems that the webrtc object is not able to actually make a connection. The latest thing I tried was manually hard-coding the turnserver url into the webrtc code, but I haven't really observed any effect.

[poljar]

Thanks for taking a shot at this.

We have two webrtc branches. One is using aiortc the other one as you noticed gstreamer.

The aiortc one successfully connects to a riot call and transmits the generated picture. Sadly as the media backend it uses doesn't use gstreamer so it's a bit harder to integrate it into an UI.

The GStreamer one can be easily integrated into a GTK based UI, but I never could make a connection to riot, note that the implementation as far as i remember did successfully work with itself (connecting two GStreamer webrtc clients together).

I assume that the GStreamer based webrtc implementation isn't fully done or correct. We would probably need to contact upstream and try to work out what's wrong with it.

[rexroni]

Do you have any strong preferences on which solution is "better"? I think I'm reasonably impressed by the gstreamer library; I'd be happy to hack away at that solution.

[poljar]

No preference at all, whatever works™. Using gstreamer would be probably better since it has so much support in the differing GUI toolkits.

[rexroni]

Well I got it Kinda Working™. I can auto-answer voice calls from riot. What were you thinking in terms of a user interface? I think it would be neat to have an interface where audio-only calls worked without any gui support, but I'm having trouble imagining how that might work.

[poljar]

That's great news.

For the user interface, nothing fancy is needed:

• hangup
• mute the microphone
• optionally if it isn't annoying to implement, disable the webcam

GUI-less audio support sounds neat, but probably hard to get right, and not everybody would be a fan of it. Lets focus first on the main use-case.

Any info on how you got it working?

[rexroni]

Well I tried a lot of things, and I'm not 100% certain which ones were critical and which ones weren't. But the important ones seem to be:

• Ignore the negoitiation-needed signal when answering, since that's intended to trigger the offerer to create-offer (not sure why the signal fires on the answerer after all the normal signaling is complete, but that's what I was seeing).
• The GST 3.0 python plugin seems to be much less pythonic than the 1.0 version... I think it's entirely autogenerated or something. Things like reply["answer"] had to be replaced with reply.get_value("answer") which is basically identical to the C api. Similarly, self.pipe.add(q, conv, sink) had to become three separate self.pipe.add() calls.
• Messages which are emitted before the pipeline is started are just dropped entirely. There was an issue where start_pipeline() was called right after set-remote-description, which was resulting in a None value for reply from that promise... which was causing havoc right before we could set the local description.
• I added better logging of errors on the matrix plugin side, especially around trying to parse messages. Looking back at the original code, there was a line that called print(self.role) to stdout which may have been preventing the plugin-side message parsing from ever working, since unparsable messages were silently assumed to be incomplete messages. So if the buffer from stdout looked like answer\n{"type": ...} no messages would ever get parsed. This was one of the last bugs I found, but by then the offending print() call was one I added.

[GideonWolfe]

Am I understanding correctly that this would allow voice chat with matrix through weechat? Just trying to understand the scope of what this plugin can currently do.

[poljar]

Well yes and no, you answer a call through Weechat and then it spawns a separate program to do the call.

[GideonWolfe]

Ok, Interesting. Would this be more "behind the scenes" or would this other program be the main way of interacting with the call?

[poljar]

The other program should spawn a window where you could see the video stream of the other participant and have some controls to mute, hide/show your camera, and end the call.

[GideonWolfe]

This is awesome, I hadn't even considered video calling, only voice. Well I certainly am looking forward to this feature!