Problem with FoVPerspectiveCameras.unproject_points

[olivierp9]

Hello,

I tried using unproject_points using this snippet to convert back depth map into a point cloud:

import numpy as np
import open3d as o3d
import torch
from pytorch3d.io import load_objs_as_meshes, load_obj
from pytorch3d.renderer import (
    FoVPerspectiveCameras, look_at_view_transform,
    RasterizationSettings, MeshRasterizer
)

width = 480
half_width = width / 2.0
height = 640
half_height = height / 2.0
fovy = 45
obj_path = './box.obj'

verts, faces, aux = load_obj(obj_path)
meshes = load_objs_as_meshes([obj_path])
meshes = meshes.cuda()

R, T = look_at_view_transform(eye=((0, -1.0, -0.8),), at=((0, 0, 0),), up=((0, 0, 1),))
R = R.cuda()
T = T.cuda()
cameras = FoVPerspectiveCameras(znear=0.5, R=R, T=T, fov=fovy)
cameras = cameras.cuda()

raster_settings = RasterizationSettings(
    image_size=(height, width),
    blur_radius=0.0,
    faces_per_pixel=1,
    # max_faces_per_bin=20000
)

rasterizer = MeshRasterizer(
    cameras=cameras,
    raster_settings=raster_settings
)

depth = rasterizer(meshes).zbuf.squeeze(-1).squeeze(0)

# so that sparse filter out depth that is equal to -1
depth[depth == -1] = 0

sparse_depth = depth.to_sparse()
indices = sparse_depth.indices()
values = sparse_depth.values()
xy_depth = torch.cat((indices.T, values[..., None]), dim=-1)
# normalize xy to [-1,1]
xy_depth[:, 0] -= half_height
xy_depth[:, 0] /= half_height
xy_depth[:, 1] -= half_width
xy_depth[:, 1] /= half_width

points = cameras.unproject_points(xy_depth)

I exported the point cloud to a ply and imported it in Meshlab and I would have expected the point cloud to match the mesh, but there seems to be transform that is not applied correctly.

Thanks in advance,

Olivier