Random long delays between depth message publishing.

[ricardoduarte27]

Preliminary Checks

• [X] This issue is not a duplicate. Before opening a new issue, please search existing issues.
• [X] This issue is not a question, feature request, or anything other than a bug report directly related to this project.

Description

When using the zed_wrapper_node to publish ZED2i topics (see configuration file attached), and having a subscriber listening to the zed_node/depth/depth_registered topic, we have verified that sometimes the subscriber stops receiving messages (or has a long delay between 2 messages, which can be up to 8 seconds) after a seemingly random time span.

We've tried to run the node with different quality settings - first with Neural, then with Ultra - but to no avail. It will sometimes run for 60 hours in a row without issues, but on other occasions we'll stop receiving messages (or have a long delay between 2 consecutive messages) after as little as 1 or 2 hours.

There is no logging from the zed node when this happens, it's as if nothing has happened - except that no messages are published for a long period of time.

zed2i_settings.zip

Steps to Reproduce

1. Run zed_wrapper_node with a ZED2i camera, and the settings provided in this issue
2. Run a simple subscriber to the zed_node/depth/depth_registered topic, which logs long delays between messages received (in our case, we were running the publisher at 15Hz, and the delays were sometimes as high as 8 seconds)
3. After sometime (could be 1 hour, could be 60 hours) there is an abnormal delay between messages received by the subscriber node

Expected Result

Publishing of depth data doesn't stop or has momentary long delay after a random time interval.

Actual Result

After a seemingly-random time interval (could be 1 hour, could be 60 hours), the subscriber doesn't get any new messages for a long period of time (as much as 8 seconds), and there is no logging on the zed wrapper side that indicates there was a problem.

ZED Camera model

ZED2i

Environment

OS: Ubuntu 22.04
CPU: Intel Core i9 10900
GPU: NVidia GeForce RTX 3060
ZED SDK version: 3.7.6
ROS: Noetic

Anything else?

No response

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment otherwise it will be automatically closed in 5 days

[ricardoduarte27]

This is a comment to keep this issue alive, as the problem is still present.

[Myzhar]

Hi @ricardoduarte27 this seems a not easy issue to be verified. Can you try to do a similar test by using one of the ZED SDK examples (i.e. Depth Sensing)? You can for example compare the timestamps of consecutive frames and issue a warning when it's "too different".

This kind of test is important to understand if it's something related to a ZED node behavior, a ROS behavior, or a ZED SDK behavior.

[ricardoduarte27]

Hi @Myzhar, thanks for your response. I'll try what you suggest and post here results as soon as I have them.

[ricardoduarte27]

As suggested, I ran the standalone depth sensing python script from the ZED examples, with the visualizer turned off while logging when the time between frames exceeded a certain threshold.

Most of the times it stays within the expected frame rate, sometimes goes a bit above (which is no problem). But there were occasions where it spiked to over 8 seconds or even over 15 seconds:

Running Depth Sensing sample ... Press 'Esc' to quit
STARTED TIMER
2022-10-19 17:45:25.995531: TOOK LONGER THAN EXPECTED 18ms: 0.03060294594615698 secs!
...
2022-10-20 01:24:04.430429: TOOK LONGER THAN EXPECTED 18ms: 8.768791195005178 secs!
...
2022-10-20 04:34:44.737793: TOOK LONGER THAN EXPECTED 18ms: 15.744906498934142 secs!
...
2022-10-20 05:59:53.022684: TOOK LONGER THAN EXPECTED 18ms: 8.740099856979214 secs!
...
2022-10-20 06:43:14.422360: TOOK LONGER THAN EXPECTED 18ms: 8.788333432981744 secs!
...
2022-10-20 07:20:01.928115: TOOK LONGER THAN EXPECTED 18ms: 8.835101215983741 secs!
2022-10-20 07:22:53.337182: TOOK LONGER THAN EXPECTED 18ms: 8.61554629507009 secs!
2022-10-20 07:40:03.825495: TOOK LONGER THAN EXPECTED 18ms: 8.832251735031605 secs!

(I've attached the full log, for reference) zed_sdk_depth_sensing.txt

Indeed it doesn't seem to be related to the ROS wrapper. Should this issue be posted somewhere else then?

[Myzhar]

Yes, please write an email to [email protected] adding also the report file generated by the ZED Diagnostic tool. I close this issue