D457 IMU timestamp jumps when connected via GMSL

[micah-cobot]

Required Info
Camera Model	D457
Firmware Version	5.13.1.53
Operating System & Version	Jetpack 5.1.1 / R35 (release), REVISION: 3.1
Kernel Version (Linux Only)	Jetpack 5.1.1 / R35 (release), REVISION: 3.1
Platform	NVIDIA Jetson Orin / ANVIL Orin
SDK Version	2.55.1
Language	N/A
Segment	Robot

Issue Description

I observe periodic large timestamp jumps between consecutive IMU messages coming from a D457 connected via GMSL. The issue does not happen when the exact same camera is instead connected via USB.

I initially saw this issue when subscribing to IMU messages coming from the realsense ROS2 wrapper. I configured the device to output accel and gyro measurements at 200Hz. The following are the realsense2_camera node params:

realsense_node_params = {
    'log_level': 'info',
    'rgb_camera.profile': '0,0,0',
    'enable_color': False,
    'enable_depth': False,
    'enable_infra': False,
    'enable_infra1': False,
    'enable_infra2': False,
    'enable_gyro': True,
    'enable_accel': True,
    'gyro_fps': 200,
    'accel_fps': 200,
    'unite_imu_method': 0, # happens also when set to 1, 2
    'enable_sync': True, # breaks when set to False
    'pointcloud.enable': False,
    'enable_rgbd': False,
    'align_depth.enable': False,
    'publish_tf': True,
    'depth_module.emitter_enabled': False,
    'depth_module.emitter_always_on': False,
    'depth_module.infra_profile': '640,480,30',
    'hold_back_imu_for_frames': False,
    'use_intra_process_comms': True # happens also when set to False
}

This is a plot of consecutive time stamp differences when connected via GMSL (timestamps coming from the IMU message ROS header):

And this is a plot of consecutive time stamp differences when connected via USB (timestamps coming from the IMU message ROS header):

As we can see

1. There is almost no jitter over USB but sometimes up to 100ms difference over GMSL.\
2. Over GMSL the rate is not actually 200 Hz but a bit slower, around 180 Hz

To check that this was not a ROS2 issue, I printed the consecutive message timestamp difference before data is published to ROS2 by looking at the data before the Publish call here:

https://github.com/IntelRealSense/realsense-ros/blob/ros2-master/realsense2_camera/src/base_realsense_node.cpp#L456

And I still saw that there were sometimes ~100ms time difference between consecutive messages.

Also, when I set enable_sync: False in the above params, I get one of these two error messages:

@MartyG-RealSense would you or @Nir-Az be able to advise on fixing this issue? Has it been observed before? This D457 has the factory fw version, and I cannot update it to the newest version while using IMU due to a different firmware updating issue which I posted about.

Hardware Info

I have the camera connected via GMSL to an Anvil Embeded System: https://connecttech.com/product/anvil-embedded-system-with-nvidia-jetson-agx-orin/ running Jetpack 5.1.1; this computer has a board for GMSL, and the D457 is supported as per this table from ConnectTech: https://connecttech.com/supported-cameras/:

The output of cat /etc/nv_tegra_release is :

# R35 (release), REVISION: 3.1, GCID: 32827747, BOARD: t186ref, EABI: aarch64, DATE: Sun Mar 19 15:19:21 UTC 2023

I installed librealsense via Debian package as per this tutorial: https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md. The output of dpkg -l | grep librealsense is:

ii  librealsense2:arm64                        2.55.1-0~realsense.3336                          arm64        Intel(R) RealSense(tm) Cross Platform API - runtime
ii  librealsense2-dev:arm64                    2.55.1-0~realsense.3336                          arm64        Intel(R) RealSense(tm) Camera Capture API - development files
ii  librealsense2-gl:arm64                     2.55.1-0~realsense.3336                          arm64        Intel(R) RealSense(tm) - GLSL-enabled extensions
ii  librealsense2-udev-rules:arm64             2.55.1-0~realsense.3336                          arm64        Intel(R) RealSense(tm) Camera Capture API - udev rules
ii  librealsense2-utils:arm64                  2.55.1-0~realsense.3336                          arm64        Intel(R) RealSense(tm) Camera Capture API - utils and demos

[MartyG-RealSense]

Hi @micah-cobot I will consult my Intel RealSense colleagues about your question. Thanks very much for your patience.

[Nir-Az]

Hi @micah-cobot , This will require investigation on our side. We do not support JP5.1.1 but we will try it with JP5.0.2 / JP5.1.2 Hopefully soon we can update, might take some time as we are about to release a new beta version.