Calculate ground truth depth

[primecai]

Dear authors,

Many thanks for open sourcing the impressive work. I came to notice that in order to calculate the depth maps, in https://github.com/sunset1995/DirectVoxGO/blob/341e1fc4e96efff146d42cd6f31b8199a3e536f7/lib/dvgo.py#L383 the depth is calculated using step_id, which returns the depth only to a scale. I'm wondering what modification is needed if we would like it to calculate the ground truth depth with the correct scale that aligns with the camera matrices? Based on the code I assume I need to perform the sum of weights * step_id * stepdist---it will be really helpful if the authors can give me some guidance on whether this is correct!

Best regards.

[JamieWatson683]

Hi - I also was interested in this.

Maybe there is a better way but I was able to recover the depth map by multiplying the ray_pts by the world2cam matrix, and taking the -z value, e.g.

        if cam2world is not None:
            world2cam = torch.linalg.inv(cam2world)[ray_id]  # N x 4 x 4
            # concat with 1s for 4x4 multiplication
            pts_homogeneous = \
                torch.cat((ray_pts, torch.ones_like(ray_pts[:, 0:1])), 1).unsqueeze(-1)
            z_val = \
                torch.matmul(world2cam, pts_homogeneous)[..., 2].squeeze(-1)  # select z component
            z_val *= -1  # openGL has z pointing towards the camera
        else:
            z_val = 1.0

        if render_kwargs.get('render_depth', False):
            # with torch.no_grad():
            depth = segment_coo(
                    src=weights * z_val,
                    index=ray_id,
                    out=torch.zeros([N]),
                    reduce='sum')
            ret_dict.update({'depth': depth})

So you just need to pass cam2world into the forward function as an extra argument (need to handle the sampling however).

Hope that helps!

[sunset1995]

A simpler way is:

n_step = segment_coo(
    src=(weights * step_id),
    index=ray_id,
    out=torch.zeros([N]),
    reduce='sum')
depth = t_min + stepdist * n_step

[rockywind]

Hi, @sunset1995 How can I caculate the ground depth in real-world scale in the dmpigo.py. I found the render depth is between 0 and 1.

I can't found the t_min and stepdist in the dmpigo.py if I use this method to caculate the depth.

depth = t_min + stepdist * n_step

[szgy66]

Hi - I also was interested in this.

Maybe there is a better way but I was able to recover the depth map by multiplying the ray_pts by the world2cam matrix, and taking the -z value, e.g.

        if cam2world is not None:
            world2cam = torch.linalg.inv(cam2world)[ray_id]  # N x 4 x 4
            # concat with 1s for 4x4 multiplication
            pts_homogeneous = \
                torch.cat((ray_pts, torch.ones_like(ray_pts[:, 0:1])), 1).unsqueeze(-1)
            z_val = \
                torch.matmul(world2cam, pts_homogeneous)[..., 2].squeeze(-1)  # select z component
            z_val *= -1  # openGL has z pointing towards the camera
        else:
            z_val = 1.0

        if render_kwargs.get('render_depth', False):
            # with torch.no_grad():
            depth = segment_coo(
                    src=weights * z_val,
                    index=ray_id,
                    out=torch.zeros([N]),
                    reduce='sum')
            ret_dict.update({'depth': depth})

So you just need to pass cam2world into the forward function as an extra argument (need to handle the sampling however).

Hope that helps!

Hi，In the case of dmpigo.py, how do you calculate the depth, thank you!

[szgy66]

Hi, @sunset1995 How can I caculate the ground depth in real-world scale in the dmpigo.py. I found the render depth is between 0 and 1.

I can't found the t_min and stepdist in the dmpigo.py if I use this method to caculate the depth.

depth = t_min + stepdist * n_step

Do you solved? Thanks!