Confusion with IoU values

[yashbhalgat]

Hi @dazinovic , following your comment here: https://github.com/dazinovic/neural-rgbd-surface-reconstruction/issues/1#issuecomment-1018759211, I used the mesh_metrics.py script to compute the evaluation metrics.

I used the meshes provided by you here.

The numbers I get for Chamfer distance and Normal Correctness somewhat match the results in the paper, but the IoU I get from this script is much higher than the IoU reported in the paper.

For example,

• For Morning apartment, I get IoU=0.9411, but in the paper IoU=0.716
• For Green room, I get IoU=0.9855, but in the paper the IoU=921.

I am not sure if I am making a mistake here. Can you please advise? Thank you.

[dazinovic]

Did you change the voxel resolution to 5 cm?

[yashbhalgat]

Hi @dazinovic, how do I change the voxel resolution? Do you mean the res variable in the compute_iou function in mesh_metrics.py?

Right now, the function looks like this:

def compute_iou(mesh_pred, mesh_target):
    res = 1.1875
    v_pred = mesh_pred.voxelized(pitch=res)
    v_target = mesh_target.voxelized(pitch=res)

    v_pred_filled = set(tuple(x) for x in v_pred.points)
    v_target_filled = set(tuple(x) for x in v_target.points)
    iou = len(v_pred_filled.intersection(v_target_filled)) / len(v_pred_filled.union(v_target_filled))
    return iou

[yashbhalgat]

I tried modifying the res variable to res=5.0, but that doesn't seem to fix the issue. With res=5.0, I get an IoU of 1.0 (which is expected, I guess?).

[yashbhalgat]

Hi @dazinovic, I was wondering if you had any comments/advice regarding my above comment(s). That would be very helpful. :)

[dazinovic]

I tried modifying the res variable to res=5.0

That would be 5 meters. I think I used a value of 0.1 there. Also don't forget to update the thresholds in the other function to get a correct F-score.

[yashbhalgat]

Thanks for the reply. I tried both res=0.05 (i.e. 5cm) and res=0.1 (as you suggested), but unfortunately that still doesn't produce the IOU reported in the paper.

• With res=0.1: I get IoU=0.4275
• With res=0.05: I get IoU=0.1847
• IOU reported in the paper is 0.716

Note that I am using the meshes provided by you here, so I was expecting that the results would match? Specifically, I am computing the IOU between the provided neural_rgbd.ply and gt_mesh_culled.ply.

Please advise. Thanks.

[dazinovic]

Are the meshes aligned? Have they been culled to just the observed areas?

[QuangHieu1704]

I faced the same problem with @yashbhalgat, that I cannot reproduce your results. Can you recheck again and help us?