Textures don't load with .gltf ?

[nickponline]

Checklist

• [X] I have searched for similar issues.
• [X] For Python issues, I have tested with the latest development wheel.
• [X] I have checked the release documentation and the latest documentation (for master branch).

Describe the issue

Apparently Open3D has support for textures in .gltf (https://github.com/isl-org/Open3D/issues/1775#issuecomment-720813575) but when I render a 3D model it appears without texture and I see the following warning in the console.

FEngine (64 bits) created at 0x7f88fe000000 (threading is enabled)
FEngine resolved backend: OpenGL
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.

Steps to reproduce the bug

import cv2
import glob
import numpy as np
import open3d as o3d

render = o3d.visualization.rendering.OffscreenRenderer(width=640, height=640)

# https://www.dropbox.com/s/b4ovqdaxvuwhkyj/house.gltf.zip?dl=0
mesh = o3d.io.read_triangle_mesh(f'house/model.gltf')
material = o3d.visualization.rendering.MaterialRecord() # Create material
render.scene.add_geometry(f'mesh', mesh, material)

cam = o3d.camera.PinholeCameraParameters()
cam.extrinsic = np.asarray([[1,0,0,0],[0,1,0,0],[0,0,1,150],[0,0,0,1]]) # Translate infront of camera by 5 unit
cam.intrinsic = o3d.camera.PinholeCameraIntrinsic(640, 640, 640, 640, 320, 320)
render.setup_camera(cam.intrinsic, cam.extrinsic)

# Render image
image = np.asarray(render.render_to_image())
image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) # Convert Open3D RGB to OpenCV BGR
cv2.imshow('image', image)
cv2.waitKey(0)

Error message

Expected behavior

Textured mesh.

Open3D, Python and System information

- Operating system: macOS 10.15
- Python version: Python 3.7.10
- Open3D version: 0.15.2+b48fef1
- System architecture: arm64
- Is this a remote workstation?: no
- How did you install Open3D?: pip
- Compiler version clang 10.0.0

Additional information

No response

[nickponline]

@germanros1987 I think you might be able to answer this.

[ShreyanshDarshan]

Could you provide the gltf file that caused the problem as well?

[nickponline]

@ShreyanshDarshan It's in the comment above: https://www.dropbox.com/s/b4ovqdaxvuwhkyj/house.gltf.zip?dl=0

[nickponline]

@ShreyanshDarshan The model is linked above, I notice that headless rendering doesn't seem to work with any GLFT files.

[lcc157]

Have you solved the problem?

[nickponline]

Nope. Doesn't work.

[errissa]

@nickponline Since GLTF models are typically complex models we have a different way of loading them. Try modifying the first 3 lines of code of your script to the following:

mesh = o3d.io.read_triangle_model(f'house/model.gltf')
render.scene.add_model(f'mesh', mesh)

With this modification I get the following result...

[lcc157]

@errissa Can you solve my problem. Thanks! [5505](5505)

[nickponline]

@errissa Unfortunately I get model now but still no texture:

Output:

Console output:

FEngine (64 bits) created at 0x7fc4df000000 (threading is enabled)
FEngine resolved backend: OpenGL
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.
[Open3D WARNING] Read geometry::Image failed: missing file extension.

Code:

import cv2
import glob
import numpy as np
import open3d as o3d

render = o3d.visualization.rendering.OffscreenRenderer(width=640, height=640)

# https://www.dropbox.com/s/b4ovqdaxvuwhkyj/house.gltf.zip?dl=0
mesh = o3d.io.read_triangle_model(f'house/model.gltf')
render.scene.add_model(f'mesh', mesh)

cam = o3d.camera.PinholeCameraParameters()
cam.extrinsic = np.asarray([[1,0,0,0],[0,1,0,0],[0,0,1,150],[0,0,0,1]]) # Translate infront of camera by 5 unit
cam.intrinsic = o3d.camera.PinholeCameraIntrinsic(640, 640, 640, 640, 320, 320)
render.setup_camera(cam.intrinsic, cam.extrinsic)

# Render image
image = np.asarray(render.render_to_image())
image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) # Convert Open3D RGB to OpenCV BGR
cv2.imshow('image', image)
cv2.waitKey(0)

Directory structure:

(dev) [nickp@m1 offscreen-render (main ✗)]$ tree
.
├── house
│   ├── model.gltf
│   ├── scene_mesh_textured_material_00_map_Kd.jpg
│   ├── scene_mesh_textured_material_01_map_Kd.jpg
│   ├── scene_mesh_textured_material_02_map_Kd.jpg
│   ├── scene_mesh_textured_material_03_map_Kd.jpg
│   ├── scene_mesh_textured_material_04_map_Kd.jpg
│   ├── scene_mesh_textured_material_05_map_Kd.jpg
│   ├── scene_mesh_textured_material_06_map_Kd.jpg
│   ├── scene_mesh_textured_material_07_map_Kd.jpg
│   ├── scene_mesh_textured_material_08_map_Kd.jpg
│   └── scene_mesh_textured_material_09_map_Kd.jpg
└── main.py

1 directory, 12 files

[errissa]

@nickponline This GLTF file appears to have the texture embedded directly in the GLTF. We recently added support for GLTF embedded textures so you will need to use the latest development version of Open3D to get this to work as expected. I generated the screenshot with the latest development version.