Video delay when using h264_nvmpi with aiortc to stream to web-browser with WebRTC

[samiamlabs]

Hi!

Thanks for adding hardware-accelerated FFmpeg support to Jetson!

I'm trying to stream bi-directional video and audio as close to real-time as possible between a robot and a web browser using WebRTC. Currently, I'm using aiortc, a pretty convenient python implementation of WebRTC that uses PyAv (PyAV is a Pythonic binding for the FFmpeg libraries) to encode and decode video frames.

I'm having some issues switching from libx264 to h264_nvmpi. The encoding itself seems to work and with my test setup. I can see the video from my webcam that is encoded by h264_nvmpi in the browser. However, it is quite delayed and seems to lag more and more behind the longer I wait.

I suspect I'm using the wrong options for the codec, but I'm not very experienced with FFmpeg and don't really know where to start when it comes to looking for the correct settings for this application.

I get this output when running the encoder:

NFO:aioice.ice:Connection(2) Check CandidatePair(('172.19.0.2', 60254) -> ('172.19.0.1', 52623)) State.IN_PROGRESS -> State.SUCCEEDED
INFO:aioice.ice:Connection(2) ICE completed
Connection state is connected
INFO:aiortc.codecs.h264:=======> Creating h264 encoder!
Opening in BLOCKING MODE
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
875967048
842091865
H264: Profile = 66, Level = 31 
NVMEDIA_ENC: bBlitMode is set to TRUE 
Opening in BLOCKING MODE
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
875967048
842091865
H264: Profile = 66, Level = 31 
NVMEDIA_ENC: bBlitMode is set to TRUE 
Opening in BLOCKING MODE
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 

Any tips on how I could get this to work? Is there documentation or examples for common configurations for nvmpi available somewhere?