SemanticScene-compatible HM3D semantics annotations missing

[SgtVincent]

Habitat-Sim version

v0.3.1

Are you using the bleeding edge version?

Yes.

❓ Questions and Help

I noticed that in the latest release v0.3.1, you have incorporated the new semantic annotations from PR #2307, which addressed issues mentioned in #1836 #2253 #2323.

However, with some efforts digging into the code, I found that the semantic annotations seemed to be loaded by a set of annotations as specified in config file

• https://github.com/facebookresearch/habitat-sim/blob/5f2bc6a0bde1da77b334022ca28e8903952f4a24/data/test_assets/dataset_tests/dataset_0/test_dataset_0.scene_dataset_config.json

• https://github.com/facebookresearch/habitat-sim/blob/5f2bc6a0bde1da77b334022ca28e8903952f4a24/data/test_assets/dataset_tests/dataset_0/semantics/dataset_test_scene.semantic_config.json

My questions is whether and when you are going to release these annotations for HM3D dataset including stages, regions, etc. If you are short-handed or this task is under-privileged, please give me some ideas about how to do this. I can create a PR for this later.

Context: I am working in my fork on a project about multi-agent system using environment 3D scene graph. Having GT semantic scene would help me a lot. FYR, https://github.com/SgtVincent/habitat-lab/tree/embodied_mas/habitat-mas/scene_graph

[aclegg3]

Hey @SgtVincent,

HM3D Semantic annotations include the object and architecture categories painted on the scan meshes. There is a notion of region ids for each object, but they have not been annotated with region names. There are also no ground truth bounding volumes for the regions.

If you wanted to add these, we support 2D poly loops with floor base height and extruded ceiling height as an abstraction. You could, for example, annotate the existing scenes in blender and add the semantic regions configs to the dataset. We may release some intermediate code and process documentation to make that process more transparent, but it hasn't been a priority recently.

[SgtVincent]

Thanks for your quick response! 🚀

You could, for example, annotate the existing scenes in blender and add the semantic regions configs to the dataset.

Going to look into this. I could also try some auto-processing scripts as you mentioned in other threads.

We may release some intermediate code and process documentation to make that process more transparent, but it hasn't been a priority recently.

I understand you have quite much stuff on your list. Look forward to your update. :)

[frank-gallagher]

Hi @aclegg3 ,

we support 2D poly loops with floor base height and extruded ceiling height as an abstraction

How is this accessed?

I am getting an empty list and 0.0 when printing region.poly_loop_points and region.extrusion_height.

EDIT: I'm not sure if this is related but the SemanticScene.get_regions_for_point() is also returning an empty list. Is there something in the config I need to change to get these to work?

ANOTHER EDIT: I have opened issue #2364 to address this question.