interceptor icon indicating copy to clipboard operation
interceptor copied to clipboard

Add PayloadToMimeType map to StreamInfo

Open 5ur3 opened this issue 6 months ago • 4 comments

Description

This issue arose during the implementation of a RED FEC decoder interceptor. A sender can announce both an original codec (Opus, in this example) and RED in the SDP as follows:

a=rtpmap:111 opus/48000/2
a=fmtp:111 minptime=10;useinbandfec=1;usedtx=1
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111

Following this announcement, the sender may or may not use RED for audio transmission. Furthermore, the sender can dynamically switch between unprotected Opus and RED-protected streams during a session.

Currently, the receiver's interceptor has no way to determine the MIME type of a received packet. The StreamInfo structure only contains the capabilities of the first negotiated codec for the media section, which is not always the codec being used for transmission.

This PR resolves this ambiguity by adding a PayloadToMimeType map to StreamInfo. This allows decoder interceptors to match a received packet's payload type to its corresponding MIME type. For instance, with this change, the RED FEC decoder interceptor can now reliably differentiate between unprotected audio packets and RED-protected audio packets.

Implementation PR

The PayloadToMimeType field is set within the pion/webrtc repository. The work is handled by the following pull request: pion/webrtc#3175

This corresponding PR is currently a draft because it depends on the changes from this PR. Once this is merged, the pion/webrtc PR will be updated with a commit that bumps its dependency on this repository to include these changes, and it can then be finalized for review.

5ur3 avatar Jul 08 '25 11:07 5ur3