[BUG]: The NVIDIA encoder and libwebrtc can interact poorly, causing a long delay with a frozen picture at the start of a stream

[doctorpangloss]

Package version

2.4.0-exp.8

Environment

* OS: Windows 11
* Unity version: 2021.3.4f1

Steps To Reproduce

1. Use a scene with a simple button and NvCodec.
2. Create an offer from the Unity scene.
3. Connect a remote visitor.
4. Try to mash the button. Observe a 3s freeze between receiving the first frame (a still picture) and the button being interactable.

Current Behavior

There is a long time between the stream starting to send picture versus the subsequent frames in the stream from the client's perspective being interactable.

Expected Behavior

When the user connects and the RTP video stream receiver is ready from the perspective of the Unity application sending video frames, it should request more keyframes because it appears something is broken about the keyframe it succeeds to send.

Anything else?

Setting keyframe interval to a small number makes the scene interactable much sooner. Then, after a short period of time, switching back to infinite goplength works.

No response

[karasusan]

@doctorpangloss I am sure that there is a bug freezing the frame after the first frame. Thank you.

[karasusan]

memo: WRS-345

[karasusan]

We are testing 3.0.0-pre.2 and this issue would be fixed the version. I guess upgrading libwebrtc is the reason.

https://github.com/Unity-Technologies/com.unity.webrtc/pull/843