[BUG] Low Latency Encoding for Streaming Application

[arthurBricq]

Hi Luxonis Team,

I have been working on low latency streaming with your hardware encoder. The goal is to stream a encoded bitstream over a network, which is not doable with with raw images. Low latency is crucial for our application.

With the raw images, I achieve a very decent latency of < 100 [ms] (same values as what is mentioned on your documentation). However, when I read the encoded bitstream and decodes it on my computer, using a GStreamer pipeline, I measure a latency always greater than 300 [ms]. To do this in GStreamer, I wrote a little C++ plugin which reads the bitstream from DepthAI and sources it inside GST using "appsrc" (I can provide you with the code to build this, of course !). The overall pipeline looks like this.

"appsrc name=dai_to_gst ! h264parse ! avdec_h264 ! videoconvert ! autovideosink"

I tried to tune a little the parameters of the VideoEncoder (the quality, the number of B Frames, the profile, the rate control mode) and I observed something quite interesting: the latency associated with a high quality stream is the same as the one of an extremely poor quality stream. It seems to indicate that the problem somehow comes from hardware encoder itself. Is is possible that the hardware encoder is not "low latency compatible" ? I saw that there are several kinds of hardware encoder architectures, some of them are made specifically for low latency, some others target more the lossless compression for better results.

Of course, it is possible that I did something wrong with GStreamer. It seems like GStreamer takes a little more time than OpenCV to display images (I measure + 70 [ms] than with OpenCV when streaming raw images). Still, it remains that there is a higher latency.

Do you have any insight on this use case ? Would you know what is the best set of settings to have the lowest possible latency when performing hardware encoding for streaming applications ?

Thanks in advance again for your help, Best, Arthur

[alex-luxonis]

I assume the camera resolution is set to 4K. Could you try to reduce the FPS a bit, to 28 or 29, and see if the latency issue persists: colorCamera.setFps(28)

[arthurBricq]

I reduced both the FPS and the resolution. I am using 1080P with 25 to 30 FPS. Sorry, forgot to mention this detail !