Convert 2D points to 3D

[sergiizquierdobas]

I have taken an image by using https://pytorch3d.org/tutorials/render_textured_meshes in order to obtain and locate some points on the image. I would like to map back this points located on the image to the 3D but I don't really find how. I have tried to use the matrixes R, T and K but I don't really know how to relate the dimension of the image selected on the RasterizationSettings.

capsule_mesh = Meshes(verts = [vertices], faces = [faces], textures=textures)
R, T = look_at_view_transform(dist = 2, elev = 1.3, azim = 0)
cameras = FoVPerspectiveCameras(R = R, T = T).to(device)
raster_settings = RasterizationSettings(image_size = size, blur_radius = 0.0, faces_per_pixel = 1)
rasterizer = MeshRasterizer(cameras = cameras, raster_settings = raster_settings).to(device)
shader = SoftPhongShader(cameras = cameras).to(device)
renderer = MeshRenderer(rasterizer, shader).to(device)
image = renderer(capsule_mesh)

I use the image to extract some information pixel information on the image and are those which I would like to know it's location on the 3D map.

extrinsic_matrix = torch.cat((R[0], T), 1) new_row = torch.Tensor([[0,0,0,1]]) new_extrinsic_matrix = torch.cat((extrinsic_matrix, new_row), 0).to(device) multiplication_matrix = torch.mm(k, new_extrinsic_matrix)

pixel_point = torch.tensor([[x_coord],
                            [y_coord],
                            [1],
                            [1]]).to(device)
```
`new_model_keypoint = torch.mm(multiplication_matrix, pixel_point.float())`

Any idea would be useful

[bottler]

I don't really understand. You can use the output of rasterizer(mesh) to see for each pixel in the image, what positions on the mesh it matches? Is that useful?