Simulcast support on Windows

[suncf2]

Dear all We add the support of simulcast on Windows. The accommplishment of this includes 2 parts.

Part 1: extention of encoder One single video encoder (MSDKVideoEncoder) contains multiple sub encoders (multiple MFXVideoSession* and MFXVideoENCODE*), so that we can create several actual encoders in InitEncode corresponding to the parameters. For every frame in Encode, firstly we encode the whole size of picture in sub encoder1. Secondly we scale the picture to a smaller size with the scale factor and the encode it with encoder2. When encoding completes, we just call OnEncodedImage with SetSpatialIndex to distinguish the simulcast id.

Part 2: extention of subscription As OWT restrict the subscription of the same stream twice, it will return an error of "The same remote stream has already been subscribed" if we simply subscribe it repeatly. Our workaround is to remove this restriction first, and then create a new RemoteStream before pcc->Subscribe is called. This new RemoteStream will be passed into pcc->Subscribe method and finally will be returned back with ConferenceSubscription ptr in the success callback of Subscribe. So the app can get a different RemoteStream ptr when the subscription returns successfully. And later app can attach a window to this stream ptr. This can solve the problem of subscription multiple times with multiple render windows.

Some code shown, we modified ConferenceClient::Subscribe method. `// For simulcast, we can subscribe the same stream multiply times, so we create a new stream(auto stream) to distinguish // simulcast streams. Because video render is attached with stream. std::shared_ptr<RemoteStream> auto_stream = std::make_shared<RemoteStream>( stream->Id(), stream->Origin(), stream->Capabilities(), stream->Settings());

  auto_stream->has_audio_ = stream->has_audio_;
  auto_stream->has_video_ = stream->has_video_;
  auto_stream->Attributes(stream->attributes_);
  auto_stream->source_.audio = stream->source_.audio;
  auto_stream->source_.video = stream->source_.video;

pcc->Subscribe( auto_stream, options, [on_success, weak_this, stream_id](std::string session_id, std::shared_ptr<RemoteStream> stream) { auto that = weak_this.lock(); if (!that) return; // map current pcc if (on_success != nullptr) { std::shared_ptr<ConferenceSubscription> cp( new ConferenceSubscription(that, session_id, stream_id, stream)); on_success(cp); } }, on_failure);`

Does anyone have some opinions about simulcast extension on Windows? We hope to do some contributions to OWT project. Any suggestions are welcome. Thank you!

[jianjunz]

Hi,

Thanks for sharing your idea on simulcast support.

JavaScript SDK supports simulcast by accepting RTCRtpTransceiver as a parameter of publish method. Similar changes could apply to C++ SDK as well. However, including RTCRtpTransceiver may introduce a lot of headers of libwebrtc. We usually create wrappers for WebRTC classes.

libwebrtc has defined SimulcastEncoderAdapter which is similar to your encoder extension. OWT depends on libwebrtc M88. It might be different from the latest version.

The subscribe restriction was removed from JavaScript SDK. MediaStream is associated with a subscription instead of a RemoteStream. Therefore, a RemoteStream can be subscribed multiple times.