Warning When Queue Gets Full

[Luxonis-Brandon]

Start with the why:

Our customers (and we) have run into this sort of issue:

Any idea why 22 frames of depth are delivered ok followed by the camera disconnecting?

The problem in this case was that some outputs of the StereoDepth node were connected to XLink queues, but no reading from those queues took place, and they filled up. And by default the queues are configured as blocking (can't accept new packets when getting full) and this could lead to the device pipeline also blocking. No crash, but will stay stuck until the host takes action (which never happened).

Simply adding a line like d.getOutputQueue("rectified_left", 8, false) has the effect of reconfiguring the queue as non-blocking (last parameter), and old packets would be overwritten when the queue gets full.

A solution here would be to not connect/link the unused outputs in the pipeline. For example if the rectified streams are not used, remove/comment out this code: stereo->rectifiedLeft.link(xoutRectifL->input); stereo->rectifiedRight.link(xoutRectifR->input); Not connecting unused outputs also allows the DepthAI device to apply some performance/memory optimizations.

So we are wondering if we report a warning in case a queue gets full (especially if set as blocking, but not only), so to help diagnose situations like this? And maybe also make the log level configurable (or disable the log), if the logs would be frequent with a certain usage.

Move to the how:

First we gather all statistics from all queues (both device and host side):

• FPS of queues
• current state (blocked or not blocked) and queue behaviour blocking / non-blocking
• number of messages in queue and maximum size

Then we can:

1. issue a warning on blocked queue if user would enable (or on by default)
2. present this data either visually or programmatically, where depthai-gui would visualize it and show the user a "in depth view" of what is happening with messages in the pipeline (their flow, ...)

Move to the what:

Implementation details are TBD at this point.

[epeters3]

Something like what's described above would be amazing, but I get the sense its not currently prioritized by the team. Is there any way today for me to get at least a partial understanding of what's going on with the queues e.g. which ones are blocked, how full they are, etc.? Any API I could use to query that data myself during runtime?

I've been having issues with frozen pipelines, and am having difficulty figuring out where the pipeline is getting congested. Using depthai_pipeline_graph I can see the FPS of each node, but they all quickly go to 0, so that's not useful for troubleshooting in this case. I can also add node.warn('') messages inside of each script node, but that only gives a little visibility into the script node. So I'm looking for a way to understand what's happening with the queues.