SFU - forwarding streams, peerconnection.ontrack event

[JestrabikR]

I am trying to create an ASP.NET Core SFU, to which users can connect and stream video (maybe also audio) both ways. In the future I also have to implement recording of the streams.

I wanted to do it somehow like that:

1. client A connects to the server using WebRTC (offer from client, server sends answer, ...)
2. (client A is sending video (audio))
3. client B sends offer to the server
4. ontrack event on client A will add track(s) to the peerconnection with client B
5. server sends back answer with client A's track(s)

But unfortunately SIPSorcery does not have ontrack event, is there any way I could achieve this without ontrack? Or do I have to work with RTPSession, if so could you please explain how that would be done?

Thank you for your help!

[ha-ves]

One way is to make the SFU a Peer, let's call it 'Peer S'. This way, you can forward on the Video Frame Level.

When Peer S OnVideoFrameReceived event fires, forward it to the selected Peer (according to your forwarding configuration) SendVideo. Some pseudo-code:

Peer_S.OnVideoFrameReceived += (ep, timestamp, sample, fmt) =>
{
    if(Peer_1.WantsToReceive)
        Peer_1.SendVideo(timestamp, sample);
};

This is just a simple example.

Of course, you'd have cons where it will recopy the whole frame instead of just directly forwarding the RTP Packets and not an end-to-end encryption.

But this also has the advantage that you can do video-level optimization for a large group call (tiling into single videostream which then gets split on the client side) which will reduce bandwidth needed on client's side.

[sipsorcery]

See this comment regarding suitability of this library for implemnting an SFU.