Same mesh for semantics and color (MP3D)

[dhruvbatra]

🚀 Feature

Use the same mesh for semantics as we do for color/texture (for MP3D dataset).

More generally, develop a standard protocol/format for information (color, semantics, instance annotation, material properties, etc) on meshes.

Motivation

As stated in our README, the semantic sensor in MP3D houses currently requires the use of additional 3D meshes with orders of magnitude more geometric complexity leading to reduced performance. This leads to a significant reduction in rendering speed when using the semantics sensor.

My understanding (from talking to @msavva) is that this is because semantic annotation was done on more detailed meshes, which made the lives of the annotators easy.

Now, we should transfer this information to the color texture meshes.

This will increase FPS and reduce memory footprint (we won't have to load 2 sets of meshes for the same env), meaning that more environments will be loadable on the same GPU.

Details/Subtleties

Semantics information will need to be copied over as additional texture to the color texture meshes. @msavva has tried direct copy (and has some code available), but it won't solve the problem because sometimes there aren't enough faces/meshes on an object/surface to just copy over the semantic information. So we'll need to create semantic textures.

This will get complicated with Basis because we don’t want to compress semantics colors.

[aclegg3]

We also want to enable programmatic setting of semantic IDs for objects added dynamically to the scene. Unifying the assets this way would make this much more straightforward.

[erikwijmans]

For added objects, this is actually reasonably straightforward as the entire drawable has the same object ID. This is what the objectId parameter in the GenericDrawable is for: https://github.com/facebookresearch/habitat-sim/blob/master/src/esp/gfx/GenericDrawable.h#L24

[mosra]

This will get complicated with Basis because we don’t want to compress semantics colors.

We don't need to use Basis for everything -- here a R8Uint format of a much smaller resolution than the color texture would suffice I think. For integer data Basis is bad choice anyway because all formats they support assume a normalized [0; 1] range.

[mathfac]

Clarifying question: @msavva how do you find correspondence between semantic and original meshes for MP3D?

[msavva]

Correspondence is mostly reasonable. There are a few issues due to differences in geometry, but I believe that this projection will actually mostly help to improve the situation (by getting rid of floating geometry artifacts in the semantic mesh).