wrong image timestamps under heavy load

[christian-rauch]

Describe the bug We encountered that images are published with wrong timestamps when the CPU or GPU is under heavy load. When this happens, the driver prints the warning Initializing or re-initializing the device to realtime offset.

To Reproduce Steps to reproduce the behaviour:

1. start driver
2. stress the PC with CPU and/or GPU load
3. log colour images into bag file
4. play back the log file and subscribe to the colour images
5. when there is motion in the scene (e.g. from left to right) you should see "jumps" while the timestamps are monotonic continuous

Expected behaviour As far as I understand, the camera has a dedicated hardware clock. I would expect that, while images may be dropped under heavy load, the timestamps and image data stays synchronised and that the timestamps increase monotonically.

Desktop (please complete the following information):

• OS: Ubuntu
• Version 20.04

Additional context

The function in question is K4AROSDevice::updateTimestampOffset: https://github.com/microsoft/Azure_Kinect_ROS_Driver/blob/c0742b9e470c9e688d796029f10cb52e1a763a4a/src/k4a_ros_device.cpp#L1492-L1525 which resets the time offset and thus causes that the image data does not have a monotonic time anymore.

Is there documentation on this behaviour? The sensor has a separate hardware clock that should make sure that image data always has a monotonic time source. From my understanding, the system and the sensor both have monotonic clocks, but with different offsets. The device_to_realtime_offset_ (the offset between the world time on the system and the local time on the sensor) should only be set once when the sensor starts, and never change. What is the purpose of resetting this offset?