Inconsistency in pred_vismask computed from Corrnet

[shubhMaheshwari]

Hi, I am trying to reproduce results for Morig, but facing issues with deforming the mesh using the point cloud motion. The deformation (specifically pred_vtx_traj ) I am getting by running evaluate/eval_tracking.py is more or less static and does not correspond to results in our_results/tracking_loss/*.npz file.

Currently I have figured out that the issue is due to pred_vismask values computed by Corrnet. After running line-45 in models/deformnet.py most of the values in pred_vismask are close to 0. Because of this the number of vis_inds is approx, 20-50. But if using pred_vismask present in our_results/tracking_loss/*.npz the number is close to 900-1000.

Even on DeformingThings4D I am facing the same issue. In the figure. left,middle and right show the input mesh, target point cloud and the target mesh configuration. Color on left mesh and middle point cloud show the correspondence whereas the right mesh shows the values of pred_vismask_color using the viridis color map (blue represents 0 and green represents 1)

[shubhMaheshwari]

We tried retraining the models but the same issue persists. @zhan-xu can you help ?

[zhan-xu]

Hi here you will need to use checkpoint "deform_p_dt_seq". Could you send me this mesh and this point cloud?

[shubhMaheshwari]

As you suggested I changed the checkpoint to deform_p_dt_seq on Line:210 in evaluate/eval_tracking.py Even after making these changes, I am getting inconsistent results.

https://user-images.githubusercontent.com/22934809/212795559-5832b17d-47b1-48b7-a14c-d68dfa4c7a63.mp4

That attached video shows ouput of different components of Morig

1. Leftmost mesh shows the output (pred_vtx_traj.npy) , middle shows the input point cloud sequence (_pts_traj.npy) and rightmost shows the ground truth (_vtx_traj.npy)
2. Color on the leftmost and middle figures shows the correspondence. Mesh color calculated from get_colored_point_cloud_feature in visualize_corr.py and point_cloud_color = mesh_colors[np.argmax(corrmat,axis=0)]. Color on the righmost mesh is from pred_vismask
3. Will trying to training for deform_p_dt_seq on step-7 the losses would tend to Nan after few epochs.
4. In the comment above the example if astra_SambaDancing from the preprocessed DeformingThings4D dataset.

The main issue seems to come from pred_vismask but we are unable to figure out why is it happening. This issue is happening for all sequences. Could this be an issue with the version of pytorch used ? Can you confirm that you are also facing this issue.