Open3D
Open3D copied to clipboard
`depth_trunc` parameter ignored when calling `o3d.geometry.PointCloud.create_from_depth_image`
Checklist
- [X] I have searched for similar issues.
- [X] For Python issues, I have tested with the latest development wheel.
- [X] I have checked the release documentation and the latest documentation (for
master
branch).
Describe the issue
When trying to compute a cloud based on a depth image, the depth_trunc
parameter of the function o3d.geometry.PointCloud.create_from_depth_image
seems to be ignored. In the point cloud, there are points at where the far plane would be, i.e. further than the specified truncation. In the code snippet below, depth_trunc
is set to far
, but also changing this to lower values, e.g. 1.0, doesn't change the resulting point cloud. Is this a mistake/misunderstanding in how I use the function or a bug?
A work around is to remove the depth_trunc
value altogether, and instead include the line depth = np.where(depth < far - 0.01, depth, 0.0)
which is currently commented out in the snippet below. However, it would be nice to use depth_trunc
instead of this "hack".
Steps to reproduce the bug
import pybullet as pb
import pybullet_data
import matplotlib.pyplot as plt
import open3d as o3d
import numpy as np
if __name__ == "__main__":
pb.connect(pb.DIRECT)
pb.setAdditionalSearchPath(pybullet_data.getDataPath())
pb.loadURDF("plane.urdf")
pb.loadURDF("table/table.urdf", [0, 0, 0])
near = 0.01
far = 4.0
width = 120
height = 100
fov_x = 60
aspect = width / height
pm = pb.computeProjectionMatrixFOV(
fov=fov_x, aspect=aspect, nearVal=near, farVal=far
)
vm = pb.computeViewMatrix((0, -2, 1.5), (0, 1, 0), (0, 0, 1))
w, h, rgb, d, mask = pb.getCameraImage(width, height, vm, pm)
rgb = rgb[:, :, :3]
# Visualize rbg image
plt.imshow(rgb)
plt.show()
# Get depth image
depth = 1.0 * near * far / (far - (far - near) * d)
# depth = np.where(depth < far - 0.01, depth, 0.0)
plt.imshow(depth)
plt.show()
# Convert to open3d
o3d_depth = o3d.geometry.Image(depth)
fx = width / (2 * np.tan(np.deg2rad(fov_x) / 2))
fy = fx / aspect
o3d_intrinsics = o3d.camera.PinholeCameraIntrinsic(
width, height, fx, fy, width / 2, height / 2
)
point_cloud = o3d.geometry.PointCloud.create_from_depth_image(
depth=o3d_depth, intrinsic=o3d_intrinsics, depth_scale=1.0, depth_trunc=far
)
o3d.visualization.draw_geometries([point_cloud])
pb.disconnect()
Error message
No response
Expected behavior
No response
Open3D, Python and System information
- Operating system: Ubuntu 20.04
- Python version: Python 3.8.10
- Open3D version: 0.17.0, also tried with 0.15.2
- System architecture: x86
- Is this a remote workstation?: no
- How did you install Open3D?: pip
- Compiler version (if built from source): n/a
Additional information
No response
Please carefully check your data, its current values (maybe units are wrong) or your code.
See sample code below to show proper clipping.
import numpy as np
import open3d as o3d
if __name__ == "__main__":
print("Read Redwood dataset")
dataset = o3d.data.SampleRedwoodRGBDImages()
rgbd_images = []
depth_raw = o3d.io.read_image(dataset.depth_paths[0])
color_raw = o3d.io.read_image(dataset.color_paths[0])
d = np.asarray(depth_raw)
non_zero_d = d[np.nonzero(d)]
print("Number of depth values less than 1 meter are ", np.count_nonzero(non_zero_d < 1000))
pcd = o3d.geometry.PointCloud.create_from_depth_image(
depth_raw,
o3d.camera.PinholeCameraIntrinsic(
o3d.camera.PinholeCameraIntrinsicParameters.PrimeSenseDefault
),
np.identity(4),
depth_scale=1000.0,
depth_trunc=1,
)
print(pcd)
pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
o3d.visualization.draw_geometries([pcd])
which gives output
Read Redwood dataset
Number of depth values less than 1 meter are 4828
PointCloud with 4828 points.
Image using depth_trunc of 1.5 meter to show chair getting clipped:-
Would you like to share this depth in a npy file: depth = 1.0 * near * far / (far - (far - near) * d)
? That will help me identify the problem. Thank you!
Thanks for looking into this!
@theNded I attached the npy file with the depth image (it's also zipped since github didn't like the file ending): depth.zip
Regarding the example of @saurabheights, your depth image seems to be in mm, which you account for using depth_scale=1000.0
. In my case, the depth image is already in m, so to my understanding, depth_scale=1.0
should be correct. Or am I missing something there?
Yes, if its already in meter, depth_scale=1.0
is fine.
I am encountering the same issue (same python and o3d versions, but on ubuntu 22.04)
had a quick look at the code, and it seems that the depth_trunc
parameter is never used by the PointCloud::CreateFromDepthImage
function if the input depth image is in float64. Will try to fix it