Open3D
Open3D copied to clipboard
Published
20 hours ago •
isl-org
isl-org
Render to image does not include defaultListSSR materials
trafficstars
Checklist
- [X] I have searched for similar issues.
- [X] For Python issues, I have tested with the latest development wheel.
- [X] I have checked the release documentation and the latest documentation (for
masterbranch).
Describe the issue
I was trying to use code based off of mouse_and_point_coord.py for selecting various geometries including point clouds and mesh. It worked great until I tried to include semitransparent materials using the defaultLitSSR shader. It was if the depth map did not record them as I would click-through defaultListSSR materials, but click on all others.
In researching the issue it seems like it may be related to https://github.com/isl-org/Open3D/issues/2940 but I do not know enough about when Filament is used.
Steps to reproduce the bug
# ----------------------------------------------------------------------------
# - Open3D: www.open3d.org -
# ----------------------------------------------------------------------------
# The MIT License (MIT)
#
# Copyright (c) 2018-2021 www.open3d.org
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
# IN THE SOFTWARE.
# ----------------------------------------------------------------------------
import numpy as np
import open3d as o3d
import open3d.visualization.gui as gui
import open3d.visualization.rendering as rendering
from lib.utilities import io_utils
# This example displays a point cloud and if you Ctrl-click on a point
# (Cmd-click on macOS) it will show the coordinates of the point.
# This example illustrates:
# - custom mouse handling on SceneWidget
# - getting a the depth value of a point (OpenGL depth)
# - converting from a window point + OpenGL depth to world coordinate
class ExampleApp:
def __init__(self, cloud):
# We will create a SceneWidget that fills the entire window, and then
# a label in the lower left on top of the SceneWidget to display the
# coordinate.
app = gui.Application.instance
self.window = app.create_window("Open3D - GetCoord Example", 1024, 768)
# Since we want the label on top of the scene, we cannot use a layout,
# so we need to manually layout the window's children.
self.window.set_on_layout(self._on_layout)
self.widget3d = gui.SceneWidget()
self.window.add_child(self.widget3d)
self.info = gui.Label("")
self.info.visible = False
self.window.add_child(self.info)
self.widget3d.scene = rendering.Open3DScene(self.window.renderer)
mat = rendering.MaterialRecord()
mat.shader = "defaultUnlit"
mat.point_size = 3 * self.window.scaling
self.widget3d.scene.add_geometry("Point Cloud", cloud, mat)
mat=rendering.MaterialRecord()
cause_problem = False
if cause_problem:
mat.shader = "defaultLitSSR"
else:
mat.shader = "defaultLit"
mat.thickness = 0.1
mat.transmission = 1.0
mat.absorption_distance = 10
mat.absorption_color = np.array([0.5, 0.5, 0.5])
mat.base_color = [1, 0., 0., 0.2]
mat.base_roughness = 0.0
mat.base_reflectance = 0.0
mat.base_clearcoat = 1.0
mat.base_clearcoat_roughness = 0.0
mesh = o3d.geometry.TriangleMesh.create_sphere()
# mesh.paint_uniform_color([1, 0, 0])
self.widget3d.scene.add_geometry("mesh", mesh, mat)
bounds = self.widget3d.scene.bounding_box
center = bounds.get_center()
self.widget3d.setup_camera(60, bounds, center)
self.widget3d.look_at(center, center - [0, 0, 3], [0, -1, 0])
self.widget3d.set_on_mouse(self._on_mouse_widget3d)
def _on_layout(self, layout_context):
r = self.window.content_rect
self.widget3d.frame = r
pref = self.info.calc_preferred_size(layout_context,
gui.Widget.Constraints())
self.info.frame = gui.Rect(r.x,
r.get_bottom() - pref.height, pref.width,
pref.height)
def _on_mouse_widget3d(self, event):
# We could override BUTTON_DOWN without a modifier, but that would
# interfere with manipulating the scene.
if event.type == gui.MouseEvent.Type.BUTTON_DOWN and event.is_modifier_down(
gui.KeyModifier.CTRL):
def depth_callback(depth_image):
# Coordinates are expressed in absolute coordinates of the
# window, but to dereference the image correctly we need them
# relative to the origin of the widget. Note that even if the
# scene widget is the only thing in the window, if a menubar
# exists it also takes up space in the window (except on macOS).
x = event.x - self.widget3d.frame.x
y = event.y - self.widget3d.frame.y
# Note that np.asarray() reverses the axes.
depth = np.asarray(depth_image)[y, x]
io_utils.save_image('~/data/image.jpg', np.asarray(depth_image)*255)
if depth == 1.0: # clicked on nothing (i.e. the far plane)
text = ""
else:
world = self.widget3d.scene.camera.unproject(
event.x, event.y, depth, self.widget3d.frame.width,
self.widget3d.frame.height)
text = "({:.3f}, {:.3f}, {:.3f})".format(
world[0], world[1], world[2])
# This is not called on the main thread, so we need to
# post to the main thread to safely access UI items.
def update_label():
self.info.text = text
self.info.visible = (text != "")
# We are sizing the info label to be exactly the right size,
# so since the text likely changed width, we need to
# re-layout to set the new frame.
self.window.set_needs_layout()
gui.Application.instance.post_to_main_thread(
self.window, update_label)
self.widget3d.scene.scene.render_to_depth_image(depth_callback)
return gui.Widget.EventCallbackResult.HANDLED
return gui.Widget.EventCallbackResult.IGNORED
def main():
app = gui.Application.instance
app.initialize()
# This example will also work with a triangle mesh, or any 3D object.
# If you use a triangle mesh you will probably want to set the material
# shader to "defaultLit" instead of "defaultUnlit".
pcd_data = o3d.data.DemoICPPointClouds()
cloud = o3d.io.read_point_cloud(pcd_data.paths[0])
ex = ExampleApp(cloud)
app.run()
if __name__ == "__main__":
main()
Run the code with and without "cause_problem" set to True.
Click on the sphere with cause_problem set to False. Notice the coord in the lower left.
Click on the sphere with cause_problem set to True. Notice that coord does not register unless the cloud is behind it, i.e. the defaultLitSSR material does not show up in the depth map but we can click-through to objects behind it.
Error message
none
Expected behavior
I expect that defaultLitSSR materials will show up in the depth map, otherwise, how can we select them with a gui?
Open3D, Python and System information
- Ubuntu 20.04
- Python version: Python 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0]
- Open3D version: 0.15.2
- System architecture: x86
- Is this a remote workstation?: no
- How did you install Open3D?: pip
- Compiler version (if built from source): NA
Additional information
No response
