About nn_correspondance in evaluation

[ymingxie]

Thanks for your nice work! I have a question about evaluation.

Pred trimed mesh (tsdf-fusion from render depth) is a two-layer geometry. Is it reasonable to compute correspondence distance between pred trimed mesh and scene0xxx_xx_vh_clean_2.ply (one-layer)?

For example, red is gt (scene0xxx_xx_vh_clean_2.ply), white is trimed mesh.

Why not using gt trimed mesh (tsdf-fusion from gt depth) as ground truth for distance computing? And I get following result by doing this:

dist1 0.093 dist2 0.077 prec 0.643 recal 0.633 fscore 0.633,

which is much better than the result in your paper.

[zmurez]

That is a very good point! I am happy to hear that this explains why the mesh metrics in the paper seem much worse than expected (and don't reflect how much better the results are compared to prior work).

I used the provided ground truth mesh because it seemed like a better protocol than using my own mesh from tsdf fusion and marching cubes (I think Scannet did some mesh post processing to improve the quality of the provided meshes). However, you are correct that this double mesh created by tsdf fusion is not comparable to the single mesh leading to much worse results.

This type of issue is exactly why I think it is better to use the provided mesh as ground truth rather than generating your own. This prevents subtle issues like these from going unnoticed and making it harder to compare different works (which each may create their own ground truth slightly differently). Having 1 standard ground truth that everyone compares to (and better yet 1 standard evaluation script) makes reproducability much better.

I think that the marching cubes extraction can be changed a bit during trimmed mesh generation to remove the double mesh. The true surface corresponds to the smooth zero crossing of the surface. The second zero crossing corresponds to the -1,1 discontinuity between the default unobserved voxels (1) and the (-1) voxels at the truncation distance behind the surface. Filtering the surface to only include zero crossings that are continuous should remove the double mesh artifact.

I am not sure when I will have time/compute to fix this, rerun the evaluations and update the paper. If you would like to help with this we can discuss?

[ymingxie]

I got single-layer mesh using http://www.open3d.org/docs/release/tutorial/pipelines/rgbd_integration.html,
And I got following result (sample frames with the interval of 10): dist1 0.100 dist2 0.083 prec 0.612 recal 0.577 fscore 0.590 l1 0.179