Rework simulcast support

[k0nserv]

Currently the way simulcast support is implemented(for ingress) is that each send encoding on the remote side results in a distinct track being created. i.e.

const transceiver = pc.addTransceiver(track, { 
  direction: 'sendonly',
  sendEncodings: [{
    { rid: 'l', active: true, maxBitrate: 250000 },
    { rid: 'm', active: true, maxBitrate: 750000 },
    { rid: 'h', active: true, maxBitrate: 1500000 },
  },  
});

// Do negotiation with webrtc-rs

Will result in on_track firing three times webrtc-rs.

This is not in line with the specification as this is logically still a single track, not three separate tracks.

Similarly when egress support is merged we should rework that to use a single egress track with send encoding being internal.

Proposed behaviour

We should bring the behaviour in line with the specification i.e. only a single track should be created, regardless of the number of sendEncodings the remote peer specified.

[algesten]

We need to decide what TrackLocal really means.

• Is it codec agnostic?

[k0nserv]

Hmm, I'm struggling to understand why that's relevant here. Can you expand on the question?

[algesten]

Looking at the code TrackLocal implementations are currently hard bound to one codec (it's passed in as argument to new).

In spec, a track is just raw data, not codec specific.

Technically one can switch sending for any of the negotiated PT – which can be different codecs.

So the question is how we see the TrackLocal abstraction.

The bind call currently returns one single codec, which goes against being agnostic.

[k0nserv]

Right, but is that relevant to simulcast? Seems like a broader concern.

@Sean-Der can you provide insight from Pion?

[algesten]

If you want to handle the different levels of simulcast as one track, we need to discuss what a track is.

Different levels could be different codecs.

[k0nserv]

Hmm, isn't the common case that the each use the same codec(payload type) just at different bitrates? Generally speaking it should be possible to send any negotiated payload type but this seems like an orthogonal concern that we just don't support atm. Am I missing something?

[algesten]

That's not how it's used in our current live server i think. We use software decoder for he lower bitrate.

Not unusual.

[algesten]

Or did I make that up? Hard to tell in my current state.

[k0nserv]

Hmm I see the confusion. This issue is about ingress currently, but TrackLocal is an egress concern. However, the issues should apply to both ingress and egress.

One can still make the choice of using a software encoder vs hardware encoder if we don't support multiple payloads. All simulcast tracks are still one codec, e.g. h264, and with rid the choice can be made to use a different decoder based on the simulcast layer, right?

[lolgesten]

If TrackLocal::bind didn't return RTCRtpCodecParameters, we could probably just save away TrackLocalContext and lazy decide which PT to use once we see the packeted data later in write_sample_with_extensions or write_rtp_with_extensions.

This would also open up for simulcast, because we could lazily let RID be a parameter that decides which PT to use.

It solves both the "I can send on whatever PT I want" and "simulcast RID" problems at the same time. What I'm not sure about is how it would work to create all stuff "beneath" TrackLocal lazily, i.e. binding senders and whatnot.

[k0nserv]

We only need to solve this for the TrackLocalSample case, right? With TrackLocalStaticRTP the user would directly created the appropriate RTP packets with correct RID and PT. It should be noted that TrackLocalStaticRTP already keeps track of several bindings via fuzzy codec matching under the hood. However, when sending it doesn't take this into account, it just overrides the payload type for each binding:

https://github.com/webrtc-rs/webrtc/blob/53020156167bf7ce385d6272bb6defa56b20c938/webrtc/src/track/track_local/track_local_static_rtp.rs#L94-L100

[algesten]

I'm unclear on whether you can have multiple "bindings" to the same peer connection. It seems a bit like it's maybe to do about sending the same track to multiple connections?