MIT-SPARK
[QUESTION] How to map semantic labels of another dataset and predictions coming from a segmentation model in Hydra?
Hello,
We want to use Hydra with ade20k dataset with its ground truth semantic labels and also prediction coming from another segmentation model, i.e., paddleseg mobile trained over ade20k 150 classes dataset. Hence we want to understand how to use that dataset in Hydra and how to map the segmentation ground truth and prediction labels based on ade20k ground truth labels to Hydra label space. We are following uhumans2 dataset example for this. There we see that Hydra\config\label_spaces<label_space_name>_label_space.yaml, Hydra\config\label_remaps<label_space_name>.yaml and Hydra-ros\hydra_ros\config\color<label_space_name>.csv files are used. We see files relevant to ade20k dataset in these folders, but we have some questions how to use them exactly.
- First question is about sub/super ids. For instance, book/bookshelf related objects are labeled as super id 2 in uhumans2 and there are multiple sub ids for the same types of objects apparently inside uhumans2, such as 5, 15 for book/bookshelfs like objects based on label_spaces data. As we understand it is a remapping of dataset specific labels to more generic labels inside Hydra. Is our understanding correct?
- If so, is there a general list of super ids? Else are the super ids we see in Hydra\config\label_remaps\uhumans2_office.yaml specific for uhumans2? Can we use the same super id mapping for ade20k? To do that which files to edit exactly? All Hydra\config\label_spaces<label_space_name>_label_space.yaml, Hydra\config\label_remaps<label_space_name>.yaml and Hydra-ros\hydra_ros\config\color<label_space_name>.csv files?
- An example of remapping based on understanding is as follows: In Hydra/config/label_spaces/ade20k_full_label_space.yaml, book is labeled as 67, while in uhumans2 books/bookshelf related objects are labeled as super id 2. So if we leave ade20k labeling as it is, would it would understand it at all in hydra while constructing the environment? Else we should treat 2 as super id for books, then books in ade20k full dataset should be mapped to 2 in remapping and .csv files? Is our understanding correct?
- In uhumans2.launch, semantic_label_remap_filepath that is Hydra\config\label_spaces<label_space_name>_label_space.yaml is called based on a condition of use_single_channel_semantic_map and it is set to false by default. So is this set to false, since uhumans2 dataset publishes semantic segmentation as rgb images by default? Then for another dataset or segmentation prediction that is in semantic map format as single channel, we should set this to True and create a <label_space_name>_label_space.yaml file accordingly?
- We want to use the whole 150 labels ade20k but we see separate label space and color .csv file for that. Then we should create label_space, label remapping and .csv file for that right?
- Also, one final question about segmentation topic: To get segmentation, according to uhumans2 dataset, we see that label_topic is subscribed to /semantic/image_raw and /tesse/seg_cam/converted/image_raw (for single channel label like semantic map?). So if we want to get the predictions or ground truth from another dataset, we should publish them to the same topics, right?
I hope our questions are clear. If we misunderstand anything, please let us know.
Best regards, Püren
Do have answers to those question you posted? I am in similar situation and your answer might be beneficial for my case
yes, I managed to use ade20k labels. I can write a detail instructions here when I find some time in the next couple of days. I hope it will help. Also, I would like to use this chance to highlight the need for clearer instructions/tutorials to use this framework from scratch for any new datasets. It is not clear at all which parameters are important, where to set what etc. I appreciate the authors for developing and maintaining such a complex framework but just want to give my feedback as I think this is a great framework that can be used in more useful ways if there are clearer instructions to use it.
yes, I managed to use ade20k labels. I can write a detail instructions here when I find some time in the next couple of days. I hope it will help. Also, I would like to use this chance to highlight the need for clearer instructions/tutorials to use this framework from scratch for any new datasets. It is not clear at all which parameters are important, where to set what etc. I appreciate the authors for developing and maintaining such a complex framework but just want to give my feedback as I think this is a great framework that can be used in more useful ways if there are clearer instructions to use it.
Thanks for your quick reply. I look forward to receiving your response. It needs to not be necessarily long, it should be enough to provide some direction and clarity. As you mentioned, the instruction regarding the usage requires a lot of improvement, not only just with this argument, but many more
Hi again,
Here is what we exactly did to use ADE20k labels. We run a segmentation model to predict segmentation masks and the model weight we used was trained over ADE20k dataset. So the predicted semantic labels are directly coming as in ADE20k dataset. We needed the following files:
- /catkin_ws/src/hydra_ros/hydra_ros/config/color/ade20k_full.csv: I attach the ade20k_full.csv that we used. I couldn't remember where we found this file. Maybe from some other Hydra related repos or in ADE20k repo... ade20k_full.csv
- /catkin_ws/src/hydra/config/label_spaces/ade20k_full_label_space.yaml: This should be already in Hydra repository.
We also altered some things in uhumans2.launch:
- You need to set use_gt_semantic to false:
<arg name="use_gt_semantics" default="false" doc="use simulator-provider semantics"/> - You need to set labelspace_name accordingly:
<arg name="labelspace_name" default="$(eval 'uhumans2_office' if arg('use_gt_semantics') else 'ade20k_full')" doc="semantic label space"/> - We also changed to topic of label_topic to the segmentation prediction coming from our segmentation model that by default comes with ADE20k label names:
<arg name="label_topic" default="/our/segmentation" unless="$(arg use_gt_semantics)"/>
I hope this helps. Please let me know if you have any question.
Bests, Püren
Hello, and thank you so much for adding this necessary piece for information. However, if I remember, hydra scene graph generation is limited to lesser label count (~40, if i am not wrong). Would that full label space (~150) be utilizable?
Good question. To be honest we don't use all the 150 labels. So not all labels in .csv may be necessary. The developers of Hydra can correct me if I am wrong but as far as I understand, the objects used are stated in /catkin_ws/src/hydra_ros/hydra_ros/config/color/ade20k_full.csv file as object_labels: [...]. So we don't use full 150 labels there.