FoundationPose
FoundationPose copied to clipboard
Tracking is easy to loss when there is large motion between nearby frames
Hello,
Thank you for your great work. I am interested in your research and am comparing the pose errors between BundleSDF and FoundationPose using my self-developed viewer.
I tested these on the HO3D dataset and found that tracking tends to be lost when there is significant motion between consecutive frames.
The first video was tested on the sequence ABF12 using your publicly released code on the HO3D dataset. As shown in the video, the pose error (ADD), indicated by the Y-axis, increases significantly at frame 729 for both BundleSDF and FoundationPose due to substantial object motion between frames 729 and 728.
https://github.com/NVlabs/FoundationPose/assets/16728958/4c32ed3b-b609-4f52-b3ef-ba76622c37fd
The second video was tested on the sequence GPMF14. Similarly, the pose error increases notably at frame 655 for both systems. There is considerable object motion between frames 655 and 654.
https://github.com/NVlabs/FoundationPose/assets/16728958/89cb528e-7c5f-4a20-bcd5-2ac8ae3268e4
So, my questions are:
Why is tracking prone to loss when there is significant motion between nearby frames? Why can't tracking be restored after it is lost in FoundationPose, even though the 3D model does not change?
Look forward your kindly reply. Thank you.
it does not look like your video is loaded in the right order. There are some jumps of events. Can you confirm?
@wenbowen123 The video came from a viewer recording. The Viewer is an offline tool developed by my self. So I can jump to any frames by the mouse or keyboard. Therefore, you can see some jumps of frames.
Additionaly, for the Viewer, it firstly loads calculated poses from the running result based on FoundationPose and BundleSDF original code. Then it calculate the pose errors (ADD) between GT poses and calculated poses. Finally, the pose errors are show on the viewer left side, and object pose in the camera on right side.
HO3D dataset removed some intermediate frames from the video, which causes unrealistic jumps in the video.
@wenbowen123 Yes, there are some missing frames in the HO3D dataset. I think these missing frames can simulate large motion scenarios. I wonder if FoundationPose and BundleSDF can work well in large motion scenarios? Thanks.
For a reasonable amount of large motion, they should work. But not for the unrealistic frame dropping like the above video, cause it no longer makes sense for using the temporal cues (tracking)
I'd also suggest to double check your data loading, since the second video (meat can) at around 00:20, the video play sequence does not look right (even with dropping).
Thanks for your kind reply.