Pion WebRTC RTP stream behavior - Media latching

[gbfarah]

I am using Pion in a server setting where clients (which may be mobile) connect to it. The clients may have frequent IP address changes during a WebRTC call..

I am noticing the following behaviour. When an clients IP address changes, the incoming stream from client to Pion server is still being serviced by webrtc peerconnection (which is great) . The outbound stream from server to client still appears to be continuing to the original IP address.

Is there a way of manipulating the peerconnection to send RTP to the IP address port on which it receives an RTP stream. In telecom speak this is referred to as Comedia / Media latching being

"The mechanism to get the Public IP/PORT for return media from the first received media packets instead of using STUN/TURN to modify the IP/PORT inside the SDP is called “Media Latching”."

From a security standpoint I don't think media latching is an issue as the cryptographic side of both endpoints is untouched and they have communicated SDPs on external secure signalling channel.

The fact that one stream (the ingress stream from client) works means we are effectively half way there.

Any feedback is appreciated

[Sean-Der]

Hey @gbfarah

You need to do a ICE Restart by default when a candidate has gone to selected it sits on the singular candidate.

I like your suggestion and I think it should totally be possible. Maybe we make this an interface and you could have any custom logic you like?

This would also satisfy the asks for people who want no ICE at all. Like this

@lisay-yan What do you think of an implementation like this? I don't think we can offer a No ICE mode (because it isn't spec compliant). However we could allow you to implement your own selection logic where you could do whatever you want!

[gbfarah]

Hi Sean, Thanks for the quick followup... I wanted to avoid an ICE restart as it entails a full SDP offer answer...

I think the simplest approach is to add a new boolean setting to Peerconnection (defaulted to false) called enableMediaLatching (true/false)....

If the setting is true the peerconnection tracks the IP port of the incoming track and dynamically switches to outbound track accordingly.

Let me know your thoughts

[Sean-Der]

I put up a proposal here I would love your thoughts @gbfarah!

I want to support your use case, but I don't think we can make it just a boolean. I am worried about security/interoperability issues. I also think users may want the same feature (but with slight differences in behavior)

I think the API suggested will cover everything. I can write all the code if you are in support of it.

[gbfarah]

Hi Sean, Thanks for the followup... From a security point of view I'm of the view there is in fact no issue with boolean ... Keep in mind DTLS negotiates a secret private key with consideration to the fingerprint as carried in the SDP (ie man in middle attack not possible) Currently pion appears to be accepting RTP from any endpoint (without enforcement of IP addresses as detected by ice). Hence why in a IP address change situation we are actually getting one way audio (inbound stream accepted. Outbound stream still going to old address). All that we are doing is making an allowance that the outbound RTP is symmetrically passed back to the a same IP and port from where legitimate packets (ie ones that successfully decrypted and the peer connection is passing up the track) came from...

In terms of your suggested solution , can you please walk me through how it addresses a mid stream IP address change. If is able to address above mentioned issue that would be great

I really appreciate your help and feedback on this.

Thanks

On Mon, Sep 25, 2023 at 4:36 AM Sean DuBois @.***> wrote:

I put up a proposal here https://github.com/pion/ice/issues/623 I would love your thoughts @gbfarah https://github.com/gbfarah!

I want to support your use case, but I don't think we can make it just a boolean. I am worried about security/interoperability issues. I also think users may want the same feature (but with slight differences in behavior)

I think the API suggested will cover everything. I can write all the code if you are in support of it.

— Reply to this email directly, view it on GitHub https://github.com/pion/webrtc/issues/2585#issuecomment-1732640120, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC4WBL3NYLMYQJHOB242BKLX4B4RDANCNFSM6AAAAAA5A2LKAU . You are receiving this because you were mentioned.Message ID: @.***>

[gbfarah]

@Sean-Der Another interesting observation. Currently pion server appears to be doing media latching in the situation where the client end SDP contains no public candidate (ie all candidates are private addresses whilst pion is in cloud) .. (this can be replicated by simply disabled stun/ice server on the client).. In other words pion/webrtc must have some code that is performing the latching mechanism today where its trust inbound stream and responds to it on the basis of where it has come from. .All we really need to have this mechanism by boolean such that latching is constantly tracked on incoming packets (not just the ones at the start of a stream)

Let me know your thoughts

[gbfarah]

@Sean-Der Any feedback on this as this has become a little urgent. We are detecting some providers such at T-Mobile changes addresses every couple of minutes (sometimes multiple times a minute). Is there a quick solution whereby the latching functionality that is mentioned in the post directly above can be run constantly.... I don't believe there is a security issue as the two ends are using symmetric key encryption by this point

[Sean-Der]

Sorry this took so long @gbfarah!

I have opened a PR for this. You will get notified of incoming STUN traffic, and you switch candidates as you wish. Would love to hear your thoughts and hopefully it gets what you need

[gbfarah]

@Sean-Der Thanks for the followup on this. In terms of feedback. I'm not sure if you saw my last comment on this thread. We were really hoping for a solution where you can prime a channel that it is running in latching mode and it dynamically latches onto streams.... Pion is already doing what we require when STUN is disabled in that it dynamically identifies and responds to an inbound stream.. What we really would like is that it continues to monitor the inbound stream and responds again if there is a network change.

Correct me if I'm wrong the proposed solution requires a signalling of new candidates (something that we would prefer not to do) as we want to keep signalling as loosely coupled as possible

Currently pion server appears to be doing media latching in the situation where the client end SDP contains no public candidate (ie all candidates are private addresses whilst pion is in cloud) .. (this can be replicated by simply disabled stun/ice server on the client).. In other words pion/webrtc must have some code that is performing the latching mechanism today where it trust inbound stream and responds to it on the basis of where it has come from. .All we really need to have this mechanism by boolean such that latching is constantly tracked on incoming packets (not just the ones at the start of a stream)

[Sean-Der]

I believe this does what you need! If you set the BindingRequestHandler and return true it will latch to the latest incoming ICE Traffic. I tested this today I moved my phone between Wifi/Cell and I had no interruption in service.

The initial media latching is happening because of Peer Reflexive Candidates. When a ICE Agent sees inbound traffic that is authenticated it will create a new candidate for it. WebRTC/ICE doesn't latch to new traffic today because when ICE gets a Selected Candidate Pair it doesn't switch by design.

We can't latch on media traffic because that makes a replay attack possible. A attacker could spoof the IP+Port and then you would start sending traffic at a unsuspecting target. ICE requires bi-directional communication. When someone sends you a Binding Request you must make a Binding Response. Just to make sure you get an ACK from the remote.

[gbfarah]

Many thanks Sean... Does this mean that the client is the forced to perform an ice regathering upon detecting a disconnection ? From a client perspective it is an operation of no value given STUN is disabled ?

By latching onto media stream the client needs to do nothing but continue streaming to the server (running pion)

In relation to replay attack, keep in mind DTLS (and underlying RTP) both incorporate measures to prevent replay as part of their protocol.

Many thanks for your efforts and follow-up

On Thu, 2 May 2024, 11:38 pm Sean DuBois, @.***> wrote:

I believe this does what you need! If you set the BindingRequestHandler and return true it will latch to the latest incoming ICE Traffic. I tested this today I moved my phone between Wifi/Cell and I had no interruption in service.

The initial media latching is happening because of Peer Reflexive Candidates. When a ICE Agent sees inbound traffic that is authenticated it will create a new candidate for it.

We can't latch on media traffic because that makes a replay attack possible. A attacker could spoof the IP+Port and then you would start sending traffic at a unsuspecting target. ICE requires bi-directional communication. When someone sends you a Binding Request you must make a Binding Response. Just to make sure you get an ACK from the remote.

— Reply to this email directly, view it on GitHub https://github.com/pion/webrtc/issues/2585#issuecomment-2090525643, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC4WBL44LNPSP5KEUGVJPXDZAI6V7AVCNFSM6AAAAAA5A2LKAWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJQGUZDKNRUGM . You are receiving this because you were mentioned.Message ID: @.***>

[Sean-Der]

Does this mean that the client is the forced to perform an ice regathering upon detecting a disconnection

Nope! The ICE Agent is constantly sending STUN Binding requests (to keep the connection alive). When the client roams it doesn't need to do anything. Pion will just see ICE traffic coming from a new IP/Port (and you can switch to it if you wish)

Relay Attack That's true. The worst the attacker could do for RTP is cause the latching to switch back and forth. However with ICE the latch can't switch until the candidate pair has been confirmed valid.

If it helps I am happy to get on a call and we can discuss details how it works/get a demo together https://siobud.com/meeting

anything I can do to help I am here!

[gbfarah]

Hi @Sean-Der
Thanks for the offer to meet. Perhaps as a starting point (and avoid wasting time) can you please provide some sample code at a peerConnection level makes use of this feature. Im a bit lost on what to do ....

Currently I have this code which kills peerConnection if ice connection fails (which I presume would need to change ?? or incorporate the new feature)

peerConnection.OnICEConnectionStateChange(func(connectionState webrtc.ICEConnectionState) {
		fmt.Printf("Connection State has changed %s \n", connectionState.String())

		if connectionState == webrtc.ICEConnectionStateFailed {
			if closeErr := peerConnection.Close(); closeErr != nil {
				logger.log(LogMessage{callId: channel.sessionID, channel: strconv.Itoa(channel.channelID), level: LogLevel_Error, payload: "createPeerConnection - webrtc.ICEConnectionStateFailed error"})
			}
		}


	})

Many thanks in advance

[gbfarah]

@Sean-Der Any feedback on how a top level user (one that only cares about peerconnections) can make use of the feature

Thanks

[Sean-Der]

Hey @gbfarah check out this test.

To easily test/see the behavior change try always returning true from SetICEBindingRequestHandler and it will always switch to the latest IP it sees (from incoming ICE traffic)

[Sean-Der]

This is a pretty niche topic, if you wanna talk about it live I would love to help https://siobud.com/meeting

anything I can im here to help

[gbfarah]

Hi @Sean-Der . Thanks for leading me in right direction. I can confirm with the "return=true", it seems to be functioning as you describe. This has however uncovered another problem. We are observing that Android devices (using native chrome webrtc implementation) are behaving well in the sense that they only communicate a new candidate pair when needed (ie if WIFI signal is strong , 5G candidate is not communicated to SetICEBindingRequestHandler. Only when user turns WIFI off or moves out of range).

On iOS devices using flutter-webrtc / safari, the implementation is very chatty with a constant candidate change every 10/15 seconds....

The pion server is handling this without issue and properly switching. However I believe some US providers may have traffic shaping for uplink traffic (results in bad / choppy voice quality).... I think we may need to reduce frequency of switching on a per call basis. Is there a way to access the peerconnection object/unique identifier for the channel in SetIceBindingRequestHandler so that instead of blindly returning true we can pass to a decision function ?

Many thanks in advance

[gbfarah]

@Sean-Der Any feedback on this