Trying to load replica_cad dataset throws 'navmesh has no navigable area` error

[RachithP]

Habitat-Sim version: v0.2.1

❓ Questions and Help

Hello,

I'm trying to extract semantic images from replica_cad scenes provided in https://aihabitat.org/datasets/replica_cad. However, when I try to load any, I encounter an error - https://gist.github.com/RachithP/45799a2b096562b2845cfe35995af2b7.

This is the script I'm using to load scenes,

        extractor = ImageExtractor(
            scene_file_path,
            img_size=(256, 256),
            output=["semantic"],
            pose_extractor_name="random_pose_extractor"
        )

Looks like the navmesh is not found at the directory it is searching in. How can we point to the right navmesh?

I tried copying all navmeshes into the same directory where the staged files are and tried to load, but I'm getting this error then - https://gist.github.com/RachithP/00397b6135b4ca92e4f4291ea9d3e35a

I also noticed that downloading the replicacad_baked_lighting dataset and interactive dataset using python -m habitat_sim.utils.datasets_download --uids replica_cad_dataset replica_cad_baked_lighting --data-path /path/to/data/ is incomplete when compared to downloading from https://dl.fbaipublicfiles.com/habitat/ReplicaCAD/ReplicaCAD_dataset_v1.5.zip and https://dl.fbaipublicfiles.com/habitat/ReplicaCAD/ReplicaCAD_baked_lighting_v1.5.zip.

This is what I have after downloading using habitat_sim.utils.datasets_download script,

$ ls replicacad_baked_lighting
total 48M
-rw-rw-r-- 1 rachithp 2.3M May  6 16:41 remake_v0_v3_sc4_staging_08.glb
drwxrwxr-x 2 rachithp 4.0K May  6 16:41 .
-rw-rw-r-- 1 rachithp 1.4K May  6 16:41 LICENSE.txt
-rw-rw-r-- 1 rachithp 5.8M May  6 16:41 remake_v0_v3_sc1_staging_19.glb
-rw-rw-r-- 1 rachithp 2.6M May  6 16:41 remake_v0_v3_sc1_staging_09.glb
-rw-rw-r-- 1 rachithp 2.1M May  6 16:41 remake_v0_v3_sc0_staging_03.glb
-rw-rw-r-- 1 rachithp 1.9M May  6 16:41 remake_v0_v3_sc0_staging_02.glb
-rw-rw-r-- 1 rachithp 2.3M May  6 16:41 remake_v0_v3_sc1_staging_04.glb
-rw-rw-r-- 1 rachithp 2.6M May  6 16:41 remake_v0_v3_sc0_staging_15.glb
-rw-rw-r-- 1 rachithp 2.6M May  6 16:41 remake_v0_v3_sc1_staging_01.glb
-rw-rw-r-- 1 rachithp 2.6M May  6 16:41 remake_v0_v3_sc1_staging_02.glb
-rw-rw-r-- 1 rachithp 2.6M May  6 16:41 remake_v0_v3_sc2_staging_13.glb
-rw-rw-r-- 1 rachithp 2.3M May  6 16:41 remake_v0_v3_sc3_staging_01.glb
-rw-rw-r-- 1 rachithp 2.3M May  6 16:41 remake_v0_v3_sc4_staging_02.glb
-rw-rw-r-- 1 rachithp 2.3M May  6 16:41 remake_v0_v3_sc3_staging_14.glb
-rw-rw-r-- 1 rachithp 2.6M May  6 16:41 remake_v0_v3_sc2_staging_14.glb
-rw-rw-r-- 1 rachithp 2.3M May  6 16:41 remake_v0_v3_sc3_staging_04.glb
-rw-rw-r-- 1 rachithp 2.3M May  6 16:41 remake_v0_v3_sc4_staging_07.glb
-rw-rw-r-- 1 rachithp 2.3M May  6 16:41 remake_v0_v3_sc4_staging_13.glb
-rw-rw-r-- 1 rachithp 2.3M May  6 16:41 remake_v0_v3_sc4_staging_12.glb
-rw-rw-r-- 1 rachithp 2.3M May  6 16:41 remake_v0_v3_sc4_staging_05.glb

The unzipped folder from https://dl.fbaipublicfiles.com/habitat/ReplicaCAD/ReplicaCAD_baked_lighting_v1.5.zip is

$ ls replica_cad_baked_lighting
total 48K
drwxrwxr-x 9 rachithp 4.0K May  6 17:31 .
drwxrwxr-x 4 rachithp 4.0K May  6 17:31 ..
drwxr-xr-x 2 rachithp 4.0K Apr 20 17:20 navmeshes_default
drwxr-xr-x 2 rachithp 4.0K Apr 20 17:17 navmeshes
drwxr-xr-x 2 rachithp 4.0K Apr 20 15:33 stages
drwxr-xr-x 4 rachithp 4.0K Apr 19 16:58 configs
-rw-r--r-- 1 rachithp 1.4K Apr 19 16:58 LICENSE.txt
-rw-r--r-- 1 rachithp 7.8K Apr 19 16:58 replicaCAD_baked.scene_dataset_config.json
drwxr-xr-x 8 rachithp 4.0K Apr 19 16:58 urdf
drwxr-xr-x 8 rachithp 4.0K Apr 19 16:58 urdf_uncompressed
drwxr-xr-x 2 rachithp 4.0K Apr 19 16:58 stages_uncompressed

This is what I have after downloading using habitat_sim.utils.datasets_download script,

$ ls replicacad
total 36K
drwxrwxr-x 6 rachithp 4.0K May  6 16:41 ..
drwxrwxr-x 7 rachithp 4.0K May  6 16:41 .
drwxrwxr-x 7 rachithp 4.0K May  6 16:41 configs
-rw-rw-r-- 1 rachithp 1.4K May  6 16:41 LICENSE.txt
-rw-rw-r-- 1 rachithp 1.3K May  6 16:41 replicaCAD.scene_dataset_config.json
drwxrwxr-x 3 rachithp 4.0K May  6 16:41 objects
drwxrwxr-x 2 rachithp 4.0K May  6 16:41 navmeshes
drwxrwxr-x 8 rachithp 4.0K May  6 16:41 urdf
drwxrwxr-x 2 rachithp 4.0K May  6 16:41 stages

The unzipped folder from https://dl.fbaipublicfiles.com/habitat/ReplicaCAD/ReplicaCAD_dataset_v1.5.zip is

$ ls replica_cad
total 44K
drwxrwxr-x 8 rachithp 4.0K May  6 17:21 .
drwxrwxr-x 8 rachithp 4.0K May  6 17:16 ..
drwxr-xr-x 2 rachithp 4.0K Apr 21 13:06 navmeshes_default
drwxr-xr-x 2 rachithp 4.0K Apr 21 13:04 navmeshes
drwxr-xr-x 6 rachithp 4.0K Apr 20 16:27 objects
drwxr-xr-x 7 rachithp 4.0K Apr 19 16:58 configs
-rw-r--r-- 1 rachithp 1.4K Apr 19 16:58 LICENSE.txt
-rw-r--r-- 1 rachithp 7.8K Apr 19 16:58 replicaCAD.scene_dataset_config.json
drwxr-xr-x 2 rachithp 4.0K Apr 19 16:58 stages
drwxr-xr-x 8 rachithp 4.0K Apr 19 16:58 urdf

What are the right scene files that should be used to extract semantic images from replica_cad via habitat?

[aclegg3]

What version of Habitat are you using? The downloader is versioned, so if you installed via conda from a previous tag instead of using nightly or building main you may not have the correct downloader version.

If you want semantics from ReplicaCAD, you are correct to use ReplicaCAD_dataset_v1.5.zip. These scenes will have semantic annotations for rigid objects only (no background or URDF semantics unfortunately). The baked scenes don't have semantics at all.

[RachithP]

What version of Habitat are you using? The downloader is versioned, so if you installed via conda from a previous tag instead of using nightly or building main you may not have the correct downloader version.

@aclegg3, I'm using v0.2.1(latest stable releas). I'm not sure if the downloader versioned with this release is supposed to have the complete information or not.

If you want semantics from ReplicaCAD, you are correct to use ReplicaCAD_dataset_v1.5.zip. These scenes will have semantic annotations for rigid objects only (no background or URDF semantics unfortunately). The baked scenes don't have semantics at all.

Thanks for the clarification! I tried using ReplicaCAD dataset, however, I'm facing issues with navmesh loading - https://gist.github.com/RachithP/45799a2b096562b2845cfe35995af2b7. When I move the navmeshes from navmeshes dir to 'stages' dir, I get a different error - https://gist.github.com/RachithP/00397b6135b4ca92e4f4291ea9d3e35a. I'm not sure what I'm doing wrong here.

Another thing I observed is that ReplicaCAD dataset has 2 navmesh directories, navmeshes and navmeshes_default. What is the difference between them?

[aclegg3]

I'm using v0.2.1(latest stable releas)

Yeah, that version will be outdated as the downloader links have changed several times. Use main/nightly.

Looks like the data_extractor.py is not updated to correctly support SceneDatasets. The _config_sim method should be setting the SimulatorConfiguration.scene_dataset_config_file. For ReplicaCAD this should point to replica_cad_dataset_1.5/replicaCAD.scene_dataset_config.json. Loading that file will setup the metadata paths to correctly link the navmeshes, stages, and objects to the scene instances so you should need to move anything.

[aclegg3]

Another thing I observed is that ReplicaCAD dataset has 2 navmesh directories, navmeshes and navmeshes_default. What is the difference between them?

These were added in 1.4 and updated in 1.5 as described on the ReplicaCAD dataset info page:

.navmesh files (in navmeshes/ directory) for every scene computed for an agent with 0.3m radius (e.g. appropriate for a Fetch robot base) and additional .navmesh files (in navmeshes_default/ directory) computed with Habitat default agent parameters for optional use.

If you want to use the default navmeshes, just rename the directories and the SceneDataset will pick up the other set: navmeshes/ -> navmeshes_fetch/ navmeshes_default/ -> navmeshes/

[RachithP]

Looks like the data_extractor.py is not updated to correctly support SceneDatasets. The _config_sim method should be setting the SimulatorConfiguration.scene_dataset_config_file. For ReplicaCAD this should point to replica_cad_dataset_1.5/replicaCAD.scene_dataset_config.json. Loading that file will setup the metadata paths to correctly link the navmeshes, stages, and objects to the scene instances so you should need to move anything

Thanks for looking into this. It looks like main branch also does not set the SimulationConfiguration.scene_dataset_config_file in _config_sim method. Is this a bug or is this SimulationConfiguration.scene_dataset_config_file expected to be set elsewhere?

[aclegg3]

Yeah, this script has not been updated to consume the SceneDataset yet. Should be a relatively small fix if you'd like to try it. Just need to pass the dataset path into the ImageExtractor constructor and set it in the _config_sim method.

If you do so, feel free to open a PR to contribute the change back to main.

[RachithP]

@aclegg3, I changed the ImageExtractor constructor and all _config_sim methods to take scene_dataset_config_file as input. When I tried with this, I get the following error - https://gist.github.com/RachithP/0843bfe759c7adf066bf4c7fa7470185.

It looks like the json is successfully loaded, but there are other issues like the paths (navmeshes) are not being resolved and .ply files are missing like for example W0510 10:51:41.257417 100632 ResourceManager.cpp:252] ::loadStage : Not loading semantic mesh - File Name : /home/rachithp/code/habitat/data/replica_cad/stages/Stage_v3_sc2_staging_semantic.ply does not exist.

I did download the dataset from https://dl.fbaipublicfiles.com/habitat/ReplicaCAD/ReplicaCAD_dataset_v1.5.zip. Am I missing something here?

[RachithP]

The changes I did to ImageExtractor and associated locations works to load mp3d dataset. So. I think the issue is with the replica_cad dataset.

I was able to dig in a bit deeper. Following the error log, I found that the navmesh being searched for is hard-coded here - https://github.com/facebookresearch/habitat-sim/blob/fc7fb11ccec407753a73ab810d1dbb5f57d0f9b9/habitat_sim/simulator.py#L216 This does not work for replica_cad dataset because there is no navmesh named in such a way.

@aclegg3, the replica_cad dataset seems to have a different set of navmeshes than the one being searched for. I'm not sure if the dataset is supposed to provide this navmesh or we should handle this case differently. Any suggestions on how to handle this would be really helpful :) Thanks for all the help so far as well!

[aclegg3]

.ply files are missing like for example W0510 10:51:41.257417 100632 ResourceManager.cpp:252] ::loadStage : Not loading semantic mesh - File Name : /home/rachithp/code/habitat/data/replica_cad/stages/Stage_v3_sc2_staging_semantic.ply does not exist.

Don't worry about this message. There are no separate semantic assets for ReplicaCAD. The only semantics are object ids encoded in the object configs.

[aclegg3]

I was able to dig in a bit deeper. Following the error log, I found that the navmesh being searched for is hard-coded here

Looks like you are using an older version of habitat-sim. We have made changes to navmesh handling since then specifically to better support dataset like ReplicaCAD. I suggest you make changes to the ImageExtractor on main branch and try again.

[RachithP]

Looks like you are using an older version of habitat-sim. We have made changes to navmesh handling since then specifically to better support dataset like ReplicaCAD. I suggest you make changes to the ImageExtractor on main branch and try again.

Thanks for the suggestion @aclegg3. I'll give it a try with the main branch.

[RachithP]

Hi @aclegg3, I tried loading the replica_cad dataset by making changes to main branch. It worked! However I now face another issue - I'm getting only blank images when I try to retrieve RGB or semantic images-, I'm attaching the log from the run here - https://gist.github.com/RachithP/7bca94bc4ad08bf2a1135cc8905d01b2.

[aclegg3]

Hey @RachithP, I don't see any issues in this log, so I don't have any advice to give. Looks like the scene was loaded and navmesh successfully computed. Then no logging except a small warning before deconstruction.

[RachithP]

Hi @aclegg3, Thanks for all the help so far. Looking at these logs, there is no error that suggests as to why the rgb images are blank. What do you suggest is the best course of action to obtain rgb and semantic information from ReplicaCAD? I was also wondering if you could provide me the version of habitat that works with ReplicaCAD, so that I can use it to generate the images I need.

[aclegg3]

I would suggest using main branch, either through conda nightly or build from source with display (not headless). Then try out ReplicaCAD with the viewer.cpp application as demonstrated by the dataset page. Use key 7 to switch between RGB, Depth, and Semantic views. If that works, then it is likely a bug in the script causing issues, not the Simulator code or assets.

[RachithP]

Hi @aclegg3 , Thanks for the suggestion. I tried your suggestion with the nightly build and habitat-viewer as mentioned here works as expected. So as a further step, I tried ReplicaCAD_quickstart.py example to see if this works as this code is easier to understand. I could make it work by changing this hard-coded value. However, I observed that nowhere in this script a scene name is passed to the settings/simulator. This makes me wonder if scene_id is used at all to load/use ReplicaCAD dataset. If not then I'm not sure how to use ReplicaCAD with Image Extraction tutorial as this class uses scene_id. I tried setting scene_id="None" as done in ReplicaCAD_quickstart.py example, but it does not work with ImageExtractor class. Any suggestions on this would be very helpful.

[RachithP]

Hi @aclegg3 , I've an update. I was able to get RGB and semantic images for ReplicaCAD using ImageExtractor class by pointing scene_id to data/replica_cad/configs/scenes/*.json files. I'm not sure if this the correct approach or a hack, but if it is the correct approach, maybe this should be documented somewhere.

To extract semantic-id to object-name mapping for the objects used in the ReplicaCAD scenes, this returns empty. Is there a specific/different way to extract this information for objects in ReplicaCAD scenes?

[aclegg3]

@RachithP, sorry for the long time without an answer, I must have missed this.

Glad to hear you addressed some of these issues.

To extract semantic-id to object-name mapping for the objects used in the ReplicaCAD scenes, this returns empty. Is there a specific/different way to extract this information for objects in ReplicaCAD scenes?

Currently, there is no text to integer id mapping unique to ReplicaCAD. You would need to provide this yourself. For example, see the category mapping for MP3D here. You could start from an existing MP3D semantic txt file such as the 17DRP5sb8fy.house file accompanying the MP3D example scene.

[mikeroberts3000]

Hi @aclegg3, can you please elaborate on the following statement?

Currently, there is no text to integer id mapping unique to ReplicaCAD. You would need to provide this yourself.

Do you mean that we would need to define some additional metadata and put it in an appropriate text file somewhere, in order to use the _generate_label_map() function in the Python ImageExtractor class?

Or do you mean that we need to somehow reverse-engineer what each integer semantic_id means in the ReplicaCAD_dataset_v1.5/configs/objects/*.json files? This interpretation seems counter-intuitive, because clearly there must be some agreed-upon mapping from the integer semantic_id values to string category labels for the objects to be annotated in the first place.

For example, the value of semantic_id in frl_apartment_refrigerator.object_config.json happens to be 67. What does this value mean in terms of a string category label? If I see an integer pixel value of 67 in a semantic segmentation image returned by the ImageExtractor class, how can I determine what 67 means?

[mikeroberts3000]

Update: I just found the ReplicaCAD_dataset_v1.5/configs/ssd/replicaCAD_semantic_lexicon.json file, which appears to contain a complete mapping from integer semantic_id values to string category labels. Yay!

[aclegg3]

@mikeroberts3000 Good find, hopefully this unblocks you. :+1:

Given that the original issue is actually navmesh related, I think we should close this issue thread. If you have further questions or find any bugs related to the semantics, feel free to open a fresh issue.