blenderproc.camera.pointcloud_from_depth() return points using coordinate with the blender world as the origin instead of the camera pose.

[Seekerzero]

Describe the issue

When generate point cloud from depth with function blenderproc.camera.pointcloud_from_depth(), the result point cloud has it value in blender world coordinate instead of the coordinate in camera pose.

Minimal code example

pose_number = 0
with open(camera, "r") as f:
    for line in f.readlines():
        line = [float(x) for x in line.split()]
        position, euler_rotation = line[:3], line[3:6]
        matrix_world = bproc.math.build_transformation_mat(position, euler_rotation)
        # convert_matrix_world = (
        #     bproc.math.change_source_coordinate_frame_of_transformation_matrix(
        #         matrix_world, ["X", "Y", "Z"]
        #     )
        # )
        # bproc.camera.add_camera_pose(convert_matrix_world)
        bproc.camera.add_camera_pose(matrix_world)
        pose_number += 1

# for i in range(pose_number):
#     bvh_tree = bproc.object.create_bvh_tree_multi_objects(objs)
#     depth = bproc.camera.depth_via_raytracing(bvh_tree, i)
#     points = bproc.camera.pointcloud_from_depth(depth, i)
#     points = points.reshape(-1, 3)

#     # using numpy to filter out nan values
#     points = points[~np.isnan(points).any(axis=1)]

#     open3d_points = o3d.geometry.PointCloud()
#     open3d_points.points = o3d.utility.Vector3dVector(points)
#     # save the point cloud
#     o3d.io.write_point_cloud(
#         os.path.join(point_cloud_output, f"point_cloud_{i}.pcd"), open3d_points
#     )

bproc.renderer.enable_depth_output(False)


data = bproc.renderer.render()
# data["depth"] = bproc.postprocessing.add_kinect_azure_noise(
#     data["depth"], data["colors"]
# )

bproc.writer.write_hdf5(point_cloud_ref_image_raw_output, data)


for i in range(pose_number):
    depth = data["depth"][i]
    points = bproc.camera.pointcloud_from_depth(depth, i)
    points = points.reshape(-1, 3)

    # using numpy to filter out nan values
    points = points[~np.isnan(points).any(axis=1)]

    open3d_points = o3d.geometry.PointCloud()
    open3d_points.points = o3d.utility.Vector3dVector(points)
    # save the point cloud
    o3d.io.write_point_cloud(
        os.path.join(point_cloud_output, f"point_cloud_{i}.pcd"), open3d_points
    )

Files required to run the code

No response

Expected behavior

When creating a point cloud from a depth image, we usually assume that the generated point cloud will use the coordinates of the camera pose instead of the world coordinate frame. However, this is not mentioned in either the tutorial or the API, which can confuse the user.

BlenderProc version

2.7.0

[cornerfarmer]

Thanks for the hint. I agree this should be made more clear in the documentation