How does the interceptors work?

[Enem-20]

I need to make the video stream as stable as possible. Default interceptors should send rtcp automatically as I know. When I start Wireshark and packet drops I see RR only and picture crashes. Maybe I do something wrong:

func NewWebrtcReceiver() *WebrtcReceiver {
	wr := &WebrtcReceiver{}
	wr.lastRemoteSdp = ""
	wr.wsUrl = "ws://signallingzur.air-link.space/wstest?name=GS001D0&partnerName=001D0"
	wr.mediaEngine = &webrtc.MediaEngine{}
	loggingFactory := logging.NewDefaultLoggerFactory()
	loggingFactory.DefaultLogLevel.Set(logging.LogLevelTrace)
	s := webrtc.SettingEngine{
		LoggerFactory: loggingFactory,
	}
	s.SetICETimeouts(5*time.Second, 30*time.Second, 5*time.Second)
	s.SetIPFilter(func(ip net.IP) bool {
		return ip.To4() != nil
	})

	opusCodec := webrtc.RTPCodecParameters{
		RTPCodecCapability: webrtc.RTPCodecCapability{
			MimeType: webrtc.MimeTypeOpus,
			RTCPFeedback: []webrtc.RTCPFeedback{
				{Type: "nack"},
				{Type: "nack", Parameter: "pli"},
				{Type: "goog-remb"},
				//{Type: "ccm", Parameter: "fir"},
			},
		}, PayloadType: 111,
	}

	wr.mediaEngine.RegisterDefaultCodecs()

	if err := wr.mediaEngine.RegisterCodec(opusCodec, webrtc.RTPCodecTypeAudio); err != nil {
		panic(err)
	}

	interceptorRegistry := &interceptor.Registry{}

	if err := webrtc.RegisterDefaultInterceptors(wr.mediaEngine, interceptorRegistry); err != nil {
		log.Fatal(err)
	}
	wr.api = webrtc.NewAPI(
		webrtc.WithSettingEngine(s),
		webrtc.WithMediaEngine(wr.mediaEngine),
		webrtc.WithInterceptorRegistry(interceptorRegistry),
	)
	wr.createPeerConnection()
	wr.establishWs()
	return wr
}

[3DRX]

Hi @Enem-20, the code you show is the receiver, is the sender also pion? Or it's libwebrtc, for example, a browser.

[3DRX]

If your receiver is pion, you probably want to use TWCC instead of REMB as congestion controller's data source. Since REMB sender is not implemented in pion, but TWCC sender is enabled in RegisterDefaultInterceptors .

[Enem-20]

Hello @3DRX , Sender uses libdatachannel. I found that nack and pli works. Server shows that, but wireshark doesn't... Anyway the issue was solved after comment RR Interceptor. I'm going to research why the RR causes the picture crashing, probably this is a bug. What about a REMB and TWCC. The server end doesn't implement any of them, this is for future bitrate regulation

[3DRX]

I see, so your problem is not that congestion controller not working, your problem is smh nack not working for a pion receiver?

[Enem-20]

Do you need data or graphs from chrome://webrtc-internals to investigate the problem?

[3DRX]

If your sender is libdatachannel and your receiver is pion, where do you get chrome://webrtc-internals from? Do you mean you swap the sender to chrome for debug?

[Enem-20]

I see, so your problem is not that congestion controller not working, your problem is smh nack not working for a pion receiver?

That's what i thought at first. the server side showed that nack arrives and resending RTP occurs. i started searching further and found that image destruction occurs when RR sending is enabled

[Enem-20]

If your sender is libdatachannel and your receiver is pion, where do you get chrome://webrtc-internals from? Do you mean you swap the sender to chrome for debug?

That's right. I already test it with chrome sender. The problem with the image falling happens there too

[3DRX]

I see, chrome://webrtc-internals stats graphs surely will help! How do you simulate packet loss, and do you have SampleBuilder in receiver?

Also, there's a known issue that might effect you which fix is still a work in progress #3063 .

[Enem-20]

I see, chrome://webrtc-internals stats graphs surely will help! How do you simulate packet loss, and do you have SampleBuilder in receiver?

Also, there's a known issue that might effect you which fix is still a work in progress #3063 .

Yeah, I saw this issue. But I don't use rtx

[Enem-20]

I haven't Sample Builder. ConfigureNack uses jitterbuffer As I know and it should sort packets. Is this right?

[3DRX]

https://github.com/pion/webrtc/blob/3e43ae91a8b6ef74619cbad921bd6a841e01ec41/interceptor.go#L59 No I don't think that's the case, ConfigureNack only do nack related stuff.

Perhaps you can provide a minimal reproduction code about how you turn rtp packets to raw video frame sample, and to decoder? I'm suspecting that without SampleBuilder or jitter-buffer-like mechanism, one single packet loss or misorder can screw up the entire video stream.

[IAMouMeng]

i have the same question, default register interceptor only handle the message but it do not try to do something. Such as when i receive PIL message, interceptor will responsed, but the stream do not reset. So, how to do receive message custom and do something