libdatachannel icon indicating copy to clipboard operation
libdatachannel copied to clipboard

Add Media Interceptor API

Open SE2Dev opened this issue 1 year ago • 3 comments

This PR introduces a new "media interceptor" API (for both C and C++). This essentially allows developers to intercept media forwarding and process it themselves, optionally allowing libdatachannel to continue forwarding the media packet afterwards. This is required for users to be able to implement transport congestion control logic, etc.

Additionally the synchronized callbacks have been refactored to allow them to provide actual return types.

SE2Dev avatar Jul 15 '22 14:07 SE2Dev

Since use cases like TWCC require sending back RTCP packets, I feel like this implements a mechanism similar to the existing MediaHandler one at transport level: currently, implementing the MediaHandler interface allows to intercept media traffic in both directions, forward/drop/process it, and possibly send an answer back, only it can only be plugged in a single track.

I might be wrong, but wouldn't a transport-wide setMediaHandler method on PeerConnection better fit the use case here?

paullouisageneau avatar Jul 15 '22 21:07 paullouisageneau

So I did some additional prototyping for a new version of the interceptor API based on the feedback you gave, but I have a few questions regarding consistency with the rest of libdatachannel. A given interceptor would need to be able to drop a message, replace a message, or pass a message on with no changes; the current MediaHandler API does allow for this via MediaHandler::incoming() returning the incoming message pointer, a new one, or a null pointer. Adding the ability to specify a media handler onto PeerConnection would support the features that we need for TWCC by calling MediaHandler::incoming() near the start of PeerConnection::forwardMedia(); I'd assume MediaHandler::outgoing() would go unused in this particular case. The issue arises, when adding a corresponding C API, that the user needs to be able to specify an arbitrary interceptor callback which needs to mirror the same functionality; when replacing a given message, however, the provided callback still needs to somehow be able to allocate a new message which then still needs to somehow be automatically freed after being converted to a message_ptr somewhere internally. Given that there are a number of ways that this can be implemented, I'm not sure what the best course of action here is regarding keeping things consistent with the rest of the C API.

Also, through discussion with @dtzxporter we've concluded that it would likely be ideal to expose a method on PeerConnection that allows directly sending messages over the PeerConnection::mDtlsTransport as a DtlsSrtpTransport. With this, users are able to send arbitrary media messages back to the PeerConnection from any context in which they'd have access to the PeerConnection (e.g. sending additional packets in the interceptor callback). I have an initial implementation for what this would look like (PeerConnection::sendMedia), but the corresponding C function currently uses the name rtcSendMedia which obviously uses a naming convention extremely similar to rtcSendMessage; the issue being that rtcSendMedia accepts a handle to a peer connection rather than to a track. Is there another name that would be more preferable?

SE2Dev avatar Jul 20 '22 15:07 SE2Dev

I've rebased the pull request on the latest release, and pushed a new version of the API which is now directly based on MediaHandler. I'm not particularly satisfied with how the C API looks, but it should work for now.

Essentially, each PeerConnection contains a MediaHandler which can be used to intercept incoming messages very early in PeerConnection::forwardMedia before conditionally continuing processing on a given message. Outgoing messages are not processed at all. The C++ API provides setMediaHandler and getMediaHandler methods on PeerConnection which essentially mirror the methods of the same names on Track.

A new MediaInterceptor class has been added to capi.cpp to automatically bridge the C API for the interceptor with the existing MediaHandler setup.

The C API provides a few new functions:

Name Description
rtcSetMediaInterceptorCallback() Allows users to set an interceptor callback (and MediaHandler for the given peer connection.
rtcCreateOpaqueMessage() Provides means for users to allocate messages of a different size in their own media interceptor callback.
rtcDeleteOpaqueMessage() Provides means to explicitly delete a message that was allocated by rtcCreateOpaqueMessage().

The rtcSetMediaInterceptorCallback() allows users to provide a media interceptor callback that should essentially return one of three values:

  1. A nullptr

    In this case forwardMedia() will drop the message without additional processing.

    This is used by interceptor callbacks that have fully processed a given message and don't require additional processing, forwarding, etc.

  2. The original data pointer that the callback received.

    In this case the original message_ptr will be returned to forwardMedia().

    This is used by interceptor callbacks which essentially either perform analysis on the incoming messages, or modify the incoming messages but don't require any reallocation.

  3. An opaque pointer (rtcMessage*) that was allocated by rtcCreateOpaqueMessage().

    In this case, the MediaInterceptor will upgrade the opaque pointer to a message_ptr before passing it to forwardMedia().

    This is used in cases where an interceptor callback would be either generating a new message (to be forwarded) from the incoming one, or modifying the incoming message in such a way that its size would need to be changed.

I'm not entirely satisfied with the opaque message logic, but it should work well enough for the time being since it would likely be used pretty rarely.

The rtcCreateOpaqueMessage() function simply allocates a new rtc::Message using new, and casts it to a rtcMessage*. On the other hand, rtcDeleteOpaqueMessage() simply casts the rtcMessage* back to an rtc::Message* before calling delete on it; this shouldn't really be needed in most cases, but I figured it would be better to provide it than not to.

The MediaInterceptor::incoming() method automatically upgrades opaque rtcMessage* pointers to message_ptr. When the message_ptr reference count reaches zero, the memory allocated by rtcCreateOpaqueMessage() will automatically be freed, removing the need for an explicit call to rtcDeleteOpaqueMessage().

SE2Dev avatar Aug 05 '22 04:08 SE2Dev