IMU samples using unite_imu_method are not suitable for visual slam to work well

[amburkoff]

We encountered the problem of missing IMU frames and their unstable frequency with the option unite_imu_method:=linear_interpolation . There is a ticket https://github.com/IntelRealSense/realsense-ros/issues/898 which talks about this problem, but apparently it has not been resolved.

I will add more details to the purpose of using the IMU. We want to run visual SLAM algorithms on the D455. But we have encountered the fact that they generally do not work well or do not work at all on IMU data from the D455 camera. At the same time, we carried out tests on the ZED2 camera and all the SLAM algorithms work well on it, since there the IMU data arrives at a stable frequency of 400hz, the gyroscope and accelerometer data are synchronized or embedded (the manufacturer does not specify this or I did not figure it out), The IMU has low noise performance.

It is also desirable that the IMU data be synchronized with the images.

I looked at the update on orbslam3 there appeared an example using realsense d435i https://github.com/UZ-SLAMLab/ORB_SLAM3/blob/master/Examples/Stereo-Inertial/stereo_inertial_realsense_D435i.cc and instructions for calibrating https://github.com/ UZ-SLAMLab/ORB_SLAM3/blob/master/Calibration_Tutorial.pdf Reading the source code, you can see that they read raw data from gyro and accel and then interpolate themselves.

We used this script to independently interpolate the d455 accelerometer and gyroscope data https://drive.google.com/file/d/1PC43g8aFK7epY6PMj1HA3fQ6BjKYiKhg/view?usp=sharing

Linear interpolate of realsense: https://youtu.be/jBJGDNUjw1o Offline interpolate scripts: https://youtu.be/3nH_XpLUstQ We can see that add the end of corridor, when visual tracking, the "linear interp" IMU behavior badly same for later lost tracking, on the contrast, the IMU with "offiline interp", the tracking looks smoothly and give the reasonable trajectory when visual was lost.

It follows from this that we need the normal operation of the unite_imu_method:=linear_interpolation method for the visual slam to work

[makolele12]

@amburkoff I've got the same issue while using the D435i with ORB_SLAM3 as well. Which offline interpolation script did you use?

[amburkoff]

@amburkoff I've got the same issue while using the D435i with ORB_SLAM3 as well. Which offline interpolation script did you use? https://drive.google.com/file/d/1PC43g8aFK7epY6PMj1HA3fQ6BjKYiKhg/view?usp=sharing

[RealSenseCustomerSupport]

---

https://github.com/IntelRealSense/realsense-ros/issues/898#issuecomment-598240169 @amburkoff, would the IMU frame drop persist, or perhaps improved, when using a different host platform as @doronhi had suggested? Thank you.

[amburkoff]

@amburkoff, would the IMU frame drop persist, or perhaps improved, when using a different host platform as @doronhi had suggested? Thank you. @RealSenseCustomerSupport IMU frames drop persist on three types of PC: zotac MAGNUS EN072080S, jetson xavier, Notebook ASUS ROG STRIX Intel Core i7-10875H NVIDIA GeForce 2070 Super 16 Gb Ubuntu 20.04 realsense-ros 2.3.2 librealsense v2.50.0

[RealSenseCustomerSupport]

---

@amburkoff, thank you for your input that the issue is shown on different host systems. Could switching from linear_interpolation to "copy" method provide a workable and reliable way to mitigate imu frame drop issue?

[amburkoff]

@RealSenseCustomerSupport Yes, it improves getting IMU frames. But the copy method does not suit us, because it degrades the accuracy of the data, which impairs the performance of the SLAM algorithms.

[RealSenseCustomerSupport]

---

@amburkoff, thank you. I've tried and checked the performance of the D455 IMU frame drops on Intel NUC i7-7567U CPU @ 3.50GHz on our side. Monitoring the the timestamps reported by the IMU, frame drops is about 1 event in ~15 seconds and 16 events in ~120 seconds. The rate of the frame drop is quite low in both cases. The linear_interpolation is enabled and only one camera is running.

$ rs-enumerate-devices -s
Device Name                  Serial Number      Firmware Version
Intel RealSense D455          026222250227        05.13.00.50

Ubuntu/Kernel = Ubuntu 20.04.1 LTS / 5.4.0-42-generic
  <arg name="gyro_fps"            default="400"/>
  <arg name="accel_fps"          default="250"/>
  <arg name="enable_gyro"        default="true"/>
  <arg name="enable_accel"        default="true"/>
  <arg name="unite_imu_method"    default="linear_interpolation"/>