depthai-ros icon indicating copy to clipboard operation
depthai-ros copied to clipboard
verified publisher icon luxonis

[BUG] IMU stops publishing in 2^32 microsecond intervals

Open mrjogo opened this issue 4 months ago • 3 comments

Describe the bug

Regularly (but not 100% of the time...), when running the camera_node for long periods of time, the IMU topic (/camera_name/imu/data) will stop publishing for 2^32 microseconds (~4295 seconds, or 71.5 minutes) and then resume. The pause always starts a multiple of 2^32 us from the previous pause, and always lasts for 2^32 us. If left long enough, it will stop and resume multiple times.

I tried to isolate the cause to only certain parameters or sets of parameters, but I wasn't able to find a pattern, mostly because it sometimes doesn't freeze for a few hours (see below for parameters that do reliably fail). It SEEMS like a "good" parameter set never freezes, and a "bad" parameter set will always (eventually) freeze, but I didn't test enough and there were plenty of other uncontrolled variables on my system to be sure.

Here is the output from a monitoring node I created:

[camera_node-1] [INFO] [1757992702.086844391] [wrist_camera]: Camera ready!
[topic_stall_monitor.py-3] [INFO] [1757992702.252753213] [topic_stall_monitor]: Subscribed to /wrist_camera/imu/data [sensor_msgs/msg/Imu]
[topic_stall_monitor.py-3] [INFO] [1757992702.278115079] [topic_stall_monitor]: /wrist_camera/imu/data: learned expected period ~ 0.002s from 10 intervals
[topic_stall_monitor.py-3] [WARN] [1758005590.166395012] [topic_stall_monitor]: /wrist_camera/imu/data: no message for 5.437s (threshold 5.000s, expected period ~0.002s)
[topic_stall_monitor.py-3] [INFO] [1758009879.722961523] [topic_stall_monitor]: /wrist_camera/imu/data: resumed after gap 4294.995s (expected ~0.002s)

I did not check if any of the image or depth topics freeze as well. The fact that it's toggling at 2^32 intervals suggests some sort of timer wraparound or overflow that's not accounted for somewhere.

Running with DEPTHAI_DEBUG=1 doesn't seem to give any additional information, as the output looks the same, whether the node is frozen or not. Here's an example of the output right around when a freeze occurs (4294.967):

[camera_node-1] [19443010D18E772700] [1.3.3] [4294.941] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4294.941] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4294.971] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4294.971] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.004] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.004] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.041] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.041] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.071] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.071] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.104] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.104] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.141] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.141] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.171] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.171] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.204] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.204] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.241] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.241] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.271] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.271] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.299] [system] [info] Memory Usage - DDR: 133.71 / 333.26 MiB, CMX: 2.38 / 2.50 MiB, LeonOS Heap: 38.15 / 81.70 MiB, LeonRT Heap: 4.42 / 39.87 MiB / NOC ddr: 1348 MB/s
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.299] [system] [info] Temperatures - Average: 58.46C, CSS: 59.36C, MSS 57.44C, UPA: 58.94C, DSS: 58.08C
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.299] [system] [info] Cpu Usage - LeonOS 73.00%, LeonRT: 31.20%

Minimal Reproducible Example

Launch camera_node with the following config.yaml and leave running for a number of hours:

/**:
  ros__parameters:
    camera:
      i_pipeline_type: Depth
      i_publish_tf_from_calibration: true
      i_tf_base_frame: wrist_camera
    pipeline_gen:
      i_enable_imu: true
    stereo:
      i_align_depth: true
      i_board_socket_id: 2
      i_right_rect_publish_topic: true
      i_left_rect_publish_topic: true
      i_output_depth: true
    imu:
      i_enable_rotation: false
      i_enable_mag: false
      i_message_type: IMU
      i_gyro_freq: 400
      i_acc_freq: 400
      i_sync_method: LINEAR_INTERPOLATE_ACCEL

Expected behavior

Topic doesn't stop publishing.

Pipeline Graph

19443010D18E772700_pipeline.json

Attach system log

  • Provide output of log_system_information.py: log_system_information.json
  • Which OS/OS version are you using? Ubuntu 24.04
  • Which ROS version are you using? 2
  • Which ROS distribution are you using ? Jazzy
  • Is depthai-ros built from source or installed from apt? apt
  • Is depthai/depthai-core library installed from rosdep or manually? rosdep
  • Please include versions of following packages - apt show ros-$ROS_DISTRO-depthai ros-$ROS_DISTRO-depthai-ros ros-$ROS_DISTRO-depthai-bridge ros-$ROS_DISTRO-depthai-ros-msgs ros-$ROS_DISTRO-depthai-ros-driver 
Package: ros-jazzy-depthai
Version: 2.30.0-1noble.20250424.110737
Priority: optional
Section: misc
Maintainer: Adam Serafin <[email protected]>
Installed-Size: 68.2 MB
Depends: libc6 (>= 2.38), libgcc-s1 (>= 4.2), libopencv-core406t64 (>= 4.6.0+dfsg), libopencv-imgproc406t64 (>= 4.6.0+dfsg), libstdc++6 (>= 13.1), libopencv-dev, libusb-1.0-0-dev, nlohmann-json3-dev, ros-jazzy-ros-workspace
Homepage: https://www.luxonis.com/
Download-Size: 24.9 MB
APT-Manual-Installed: no
APT-Sources: http://packages.ros.org/ros2/ubuntu noble/main amd64 Packages
Description: DepthAI core is a C++ library which comes with firmware and an API to interact with OAK Platform

Package: ros-jazzy-depthai-ros
Version: 2.11.2-1noble.20250814.103406
Priority: optional
Section: misc
Maintainer: Adam Serafin <[email protected]>
Installed-Size: 44.0 kB
Depends: ros-jazzy-depthai, ros-jazzy-depthai-bridge, ros-jazzy-depthai-descriptions, ros-jazzy-depthai-examples, ros-jazzy-depthai-filters, ros-jazzy-depthai-ros-driver, ros-jazzy-depthai-ros-msgs, ros-jazzy-ros-workspace
Download-Size: 7,200 B
APT-Sources: http://packages.ros.org/ros2/ubuntu noble/main amd64 Packages
Description: The depthai-ros package

Package: ros-jazzy-depthai-bridge
Version: 2.11.2-1noble.20250814.084604
Priority: optional
Section: misc
Maintainer: Adam Serafin <[email protected]>
Installed-Size: 604 kB
Depends: libc6 (>= 2.38), libgcc-s1 (>= 3.3.1), libopencv-calib3d406t64 (>= 4.6.0+dfsg), libopencv-core406t64 (>= 4.6.0+dfsg), libopencv-imgcodecs406t64 (>= 4.6.0+dfsg), libopencv-imgproc406t64 (>= 4.6.0+dfsg), libstdc++6 (>= 13.1), libboost-dev, libopencv-dev, ros-jazzy-camera-info-manager, ros-jazzy-composition-interfaces, ros-jazzy-cv-bridge, ros-jazzy-depthai, ros-jazzy-depthai-ros-msgs, ros-jazzy-ffmpeg-image-transport-msgs, ros-jazzy-image-transport, ros-jazzy-rclcpp, ros-jazzy-robot-state-publisher, ros-jazzy-ros-environment, ros-jazzy-sensor-msgs, ros-jazzy-std-msgs, ros-jazzy-stereo-msgs, ros-jazzy-tf2, ros-jazzy-tf2-geometry-msgs, ros-jazzy-tf2-ros, ros-jazzy-vision-msgs, ros-jazzy-xacro, ros-jazzy-ros-workspace
Download-Size: 174 kB
APT-Sources: http://packages.ros.org/ros2/ubuntu noble/main amd64 Packages
Description: The depthai_bridge package

Package: ros-jazzy-depthai-ros-msgs
Version: 2.11.2-1noble.20250814.080551
Priority: optional
Section: misc
Maintainer: Adam Serafin <[email protected]>
Installed-Size: 2,730 kB
Depends: libc6 (>= 2.14), libgcc-s1 (>= 3.3.1), libpython3.12t64 (>= 3.12.1), libstdc++6 (>= 11), ros-jazzy-fastcdr, ros-jazzy-builtin-interfaces, ros-jazzy-geometry-msgs, ros-jazzy-rclcpp, ros-jazzy-rosidl-default-generators, ros-jazzy-sensor-msgs, ros-jazzy-std-msgs, ros-jazzy-vision-msgs, ros-jazzy-ros-workspace
Download-Size: 204 kB
APT-Sources: http://packages.ros.org/ros2/ubuntu noble/main amd64 Packages
Description: Package to keep interface independent of the driver

Package: ros-jazzy-depthai-ros-driver
Version: 2.11.2-1noble.20250814.101717
Priority: optional
Section: misc
Maintainer: Adam Serafin <[email protected]>
Installed-Size: 3,525 kB
Depends: libc6 (>= 2.38), libconsole-bridge1.0 (>= 1.0.1+dfsg2), libgcc-s1 (>= 3.3.1), libopencv-core406t64 (>= 4.6.0+dfsg), libopencv-imgproc406t64 (>= 4.6.0+dfsg), libstdc++6 (>= 13.1), libtinyxml2-10 (>= 10.0.0), ros-jazzy-ament-cmake-auto, ros-jazzy-camera-calibration, ros-jazzy-cv-bridge, ros-jazzy-depthai, ros-jazzy-depthai-bridge, ros-jazzy-depthai-descriptions, ros-jazzy-depthai-examples, ros-jazzy-depthai-ros-msgs, ros-jazzy-diagnostic-msgs, ros-jazzy-diagnostic-updater, ros-jazzy-ffmpeg-image-transport-msgs, ros-jazzy-image-pipeline, ros-jazzy-image-transport, ros-jazzy-image-transport-plugins, ros-jazzy-pluginlib, ros-jazzy-rclcpp, ros-jazzy-rclcpp-components, ros-jazzy-sensor-msgs, ros-jazzy-std-msgs, ros-jazzy-std-srvs, ros-jazzy-vision-msgs, ros-jazzy-ros-workspace
Download-Size: 750 kB
APT-Sources: http://packages.ros.org/ros2/ubuntu noble/main amd64 Packages
Description: Depthai ROS Monolithic node.

Additional context

  • Camera is OAK-D-SR
  • ros2 bag was also running and collecting the IMU data (which is how I first noticed the gap)

mrjogo avatar Sep 16 '25 16:09 mrjogo