Rendering Instance Masks

[shubham-goel]

Hi! I have multiple meshes forming a single scene. From these meshes, I want to render an instance mask containing the index of the mesh visible.

Here's an example of what I mean:

The scene is composed of different meshes (one for each object). The image on the left is an RGB image rendered from the scene, and the one on the right is an instance mask I where I(x,y) tells which object/mesh is visible at pixel (x,y). Here, the bed and guitar have different instance values.

I feel this should be possible by creating a custom texture for each object/mesh, but I'm not sure how exactly to do this. I'm currently rendering each object/mesh individually and comparing depth to create the instance mask but that's suboptimal.

Questions:

1. Can pyrender let me create such instance masks directly?
2. If yes, how do I do it?

Such masks are fairly common for many computer vision tasks and I'm sure there are others wanting to do the same. I'd appreciate any help you can offer!

Thanks, Shubham

[mmatl]

Hi! The standard way to render instance masks actually is to render individual depth images and do a depth difference between each depth image and the overall image, as you've been doing so far. In principle, it's possible to render instance masks in a single pass if you do totally flat shading and give each mesh a unique color, but for now, pyrender doesn't support totally flat shading. I can add it in as an option if you think it'd be useful, though. I haven't been having performance issues with the depth differencing method myself :)

[mmatl]

Here's an efficient way to do it all in the same renderer!

# Assume we have an offscreen renderer and a scene
segimg = np.zeros((height, width), dtype=np.uint8)
flags = RenderFlags.DEPTH_ONLY

# Render full depth
full_depth = renderer.render(scene, flags=flags)

# Hide all mesh nodes
for mn in scene.mesh_nodes:
    mn.mesh.is_visible = False

# Now, iterate through them, enabling each one
for i, node in enumerate(scene.mesh_nodes):
    node.mesh.is_visible = True
    depth = renderer.render(scene, flags=flags)
    mask = np.logical_and(
        (np.abs(depth - full_depth) < 1e-6), np.abs(full_depth) > 0
    )
    segimg[mask] = i + 1
    node.mesh.is_visible = False

# Show all meshes again
for mn in scene.mesh_nodes:
    mn.mesh.is_visible = True  

This bit of code will produce a segimg containing what you want. Does that work for you, or do you still want flat shading?

[shubham-goel]

@mmatl Thanks a lot! This is helpful but may not be enough.

I'm trying to render instance masks for video sequences from a big 3D dataset containing 1500 scenes. This roughly amounts to a total of 2.5 million images.

If I have to render each frame 20-30 times (once for each object/mesh), it seems (based on a few experimental runs) the whole thing might take up to a month. With flat-shading, however, it would be 1-2 days which is more reasonable.

It would be good to have a flat shading option. I'd be happy to help you with the implementation with some guidance!

[mmatl]

@shubham-goel Hmm after thinking about this for a bit, I think that there may be a better way to do this. It's probably better to write a custom vertex + fragment shader and just give you the option to use those in the rendering flags. Let me try to mock that up real fast.

Going to be out and about during the day today, but I should be able to get something to you tonight or tomorrow.

[mmatl]

Also, do run multiple processes at once! Your GPU/CPU should be able to handle it, and this sort of task is definitely easily parallelizable.

[arsalan-mousavian]

@shubham-goel Hmm after thinking about this for a bit, I think that there may be a better way to do this. It's probably better to write a custom vertex + fragment shader and just give you the option to use those in the rendering flags. Let me try to mock that up real fast.

Going to be out and about during the day today, but I should be able to get something to you tonight or tomorrow.

Thanks for the awesome library. Did you add this feature? If so, could you please share an example for that?

[mmatl]

Sorry y'all! Got sidetracked at work. Will try to push this out soon. It should be pretty easy to write a shader to do exactly this in a single forward pass.

[nubertj]

HI @mmatl! Thanks a lot for this cool library :) Did you have time already to look into this issue again?

[submagr]

Just found out that the instance segmentation is supported since version 0.1.41.

You can also find an example in the example.py

TL;DR:

from pyrender import RenderFlags
nm = {node: 20*(i + 1) for i, node in enumerate(scene.mesh_nodes)}   # Node->Seg Id map
seg = r.render(scene, RenderFlags.SEG, nm)[0]

[ankuPRK]

Just found out that the instance segmentation is supported since version 0.1.41.

You can also find an example in the example.py

TL;DR:

from pyrender import RenderFlags
nm = {node: 20*(i + 1) for i, node in enumerate(scene.mesh_nodes)}   # Node->Seg Id map
seg = r.render(scene, RenderFlags.SEG, nm)[0]

This did the job for me, thanks for sharing!