facebookresearch
IMU video sync issues
We've been looking carefully at the IMU data, and sometimes it seems like there are videos where the IMU/video syncing gets off a bit. In other words, just looking at the raw data, it looks like the timestamps of the IMU datapoints (which are reported relative to a start time for the video stream) are sometimes -- not always, but for portions of some videos -- misaligned to the video by at least 500ms or maybe even as much as 1s.
This may not seem like a lot in absolute terms, but it's actually quite an issue, since this will often mean that an action, such as the viewer turning their head, will appear in the video stream meaningfully before the IMU stream registers the action, breaking the temporal causality in the ordering between the streams.
I've attached a list of four videos with timestamps / frame ranges where we see the issue, but happy to give more examples if that would be helpful. It happens pretty frequently (like maybe half the times where there is IMU data to start with).
It does seem like this situation mostly (or maybe exclusively) occurs when another known problem in the data happens -- namely a non-monotonicity of the IMU data timestamps. (We read in the data documentation about this non monotonicity being a known problem.) It seems to us that whatever caused the non-monotonicity might sometimes coincide with an event that causes the IMU timestamping to get out of sync with the video timestamps.
So our questions are:
(1) Have you folks noticed this issue?
(2) Do you have idea how to fix the problem? The data documentation suggests "sorting" IMU data according to the timestamps whenever the non-monotonicity arises, but it doesn't seem like doing that would resolve the issue we're reporting. Any suggestions?
Here is a list of some of a few examples (with video ID, time, and frame number) where we saw the issue, but there are many more like this.
video_id time frame b148c2bd-349a-4c3d-af5c-6791c300bea9 01:52-01:57 3360-3510 1246d6ec-5620-4f71-8b4b-d823775f58c2 00:04-00:05 120-150 b5410470-6cb6-43ea-8233-4824ba6a27b0 10:01-10:02 18030-18060 fe058f9c-d653-488f-897e-125040d12cf6 05:27-05:29 9810-9870
