Switching video source to a different resolution/aspect ratio

[robcar]

We are trying to switch video source (IVideoSource) on a live MediaSteamTrack (SendOnly) in an open and connected RTCPeerConnection and the two video sources have different aspect ratios.

The receiving side which is displaying the video is a Chrome web browser.

If we just switch sources, on the sending SipSorcery side, the receiving browser side is displaying the new video but in the original resolution with additonal digital noise. So if the first video is 512x1024 and the second video is 512x512 the second video after the switch shows the complete image (smaller) but the bottom 512 pixels are noise.

We have tried making the video source (we are using FFmpegVideoEncoder to encode) send a KeyFrame using ForceKeyFrame() directly on the encoder but the webrtc receiver still displays the noise at the bottom.

We also tried to recreate the track and renegotiate by sending offer/answers but then the MediaStream gets muted on the receiving end in the browser.

Is there a recommended approach for this or any pointers on how to approach this, it would be greatly appreciated?

[ChristopheI]

I don't reproduce this kind of problem. I successfully switching of video stream without even the need to update SDP. I'm using VP8 as codec. Which codec are you using ?

[robcar]

Thanks for testing, that is very encouraging, just need to find out what we are doing differently :)

We are also using the VP8 codec.

And there is nothing in particular that you do when you switch the VideoSource?
No forcing of key frame? No replacing the track? Just hooking up the other VideoSource? (pseudo code:)

videoSource.SetVideoSourceFormat(selectedVideoFormat); // VP8
await videoSource.StartVideo();
videoSource.OnVideoSourceEncodedSample += videoStream.SendVideo; 

And is it ok to just unhook the old videoSource? (We might switch back to this source again)

oldVideoSource.OnVideoSourceEncodedSample -= videoStream.SendVideo; 

[Todilo]

@ChristopheI I have the same problem and I made a simple example to showcase this: https://github.com/Todilo/sipsorcery/tree/Problem_changing_aspect_ratio

basically I have 2 byte array in the same video source. On a 5 second timer I change between the two images. One has a resolution of 640x480 and the other one 500x500 . (I add the required codecs to the bin folder manually)

[ChristopheI]

I didn't have time to check/test your sample yet. But I can share with you a small example of multiplexing several video streams into one (which therefore has its size changed without pb) - video here