Docs: `map_by` in semantic segmentation is confusing

[jatentaki]

Describe the bug This is more of a documentation issue than a "hard" bug, so please excuse no self-contained repro case. I can produce one if you deem it crucial to investigating the issue.

My goal is to output semantic masks on a custom is_building attribute (aside RGB and depth, omitted for brievity). The code that works, obtained via trial and error is

objs = bproc.loader.load_blend(blend_file)

# ... omitted ...

for obj in objs:
    if 'building' in obj.get_name():
        obj.set_cp('is_building', 1)
 
data = bproc.renderer.render()
data.update(
    bproc.renderer.render_segmap(
        map_by=[
            'instance',
            'cp_is_building',
        ],
        default_values=dict(cp_is_building=0),
    )
)

bproc.writer.write_hdf5("output/", data)                                                                                                                                                                       

Discovering this incantation took me quite a moment. The example states the following:

This module can map any kind of object related information to an image or to a list of indices of the objects in the scene. So, if you want to map the custom property category_id to an image, you write map_by=["class"]. Then each pixel gets assigned the category_id of the object present in that pixel.

However, I still have three problems after reading that.

1. I do not understand where the implication made by the second sentence comes from - what is the relation between category_id and class? Let's say I want to render is_building, which map_by should I specify then?
2. Empirically, the working combination is ['instance', 'cp_is_building']. This I do not understand either - this code generates both the is_building segmentation mask and an instance segmentation mask, the latter of which I do not care about. But if I remove 'instance', the script will fail with Exception: There were attributes specified in the may_by, which could not be saved as there was no "instance" may_by key used. This is true for this/these keys: cp_is_building. Why is it necessary to run instance segmentation to enable outputting orthogonal, class-like segmentation tasks?
3. If I swap instance for class I get a surprising error message: Exception: The obj: map_18.osm_buildings does not have the attribute: cp_category_id, striped: category_id. Maybe try a default value.. But this is actually the object I explicitly set that parameter for! How come it fails to find it?

May I ask for clarification on how to understand this API? Thanks in advance!

General Information

1. Which BlenderProc version are you using? 2.3.0

2. On which operating system are you? macOS

3. Have you checked the issue tracker to see if a similar issue has been opened? Yes

4. Have you changed BlenderProc in any way besides the config file? If yes, are you sure that this change does not affect the problem you are having? I haven't changed it.

[themasterlink]

Hey,

you are absolutely right; the documentation is confusing. We should improve this.

To answer your questions:

I do not understand where the implication made by the second sentence comes from - what is the relation between category_id and class? Let's say I want to render is_building, which map_by should I specify then?

The problem is that category_id is a special one, which is not stated in the documentation. So, usually all custom properties can be rendered and then mapped to a pixel value. We could change that and allow that one could say category_id instead of class.

Empirically, the working combination is ['instance', 'cp_is_building']. This I do not understand either - this code generates both the is_building segmentation mask and an instance segmentation mask, the latter of which I do not care about. But if I remove 'instance', the script will fail with Exception: There were attributes specified in the may_by, which could not be saved as there was no "instance" may_by key used. This is true for this/these keys: cp_is_building. Why is it necessary to run instance segmentation to enable outputting orthogonal, class-like segmentation tasks?

The reason here is how these attributes are stored, most attributes are returned as dict mapping a certain key to the instance image, so you need an instance image to later on than decide for each pixel what value it has.

If I swap instance for class I get a surprising error message: Exception: The obj: map_18.osm_buildings does not have the attribute: cp_category_id, striped: category_id. Maybe try a default value.. But this is actually the object I explicitly set that parameter for! How come it fails to find it?

Have you set it like this: obj.set_cp("category_id", 1)? Can you show the code please.

Best, Max

[jatentaki]

My bad, on the last point I misread the error message as cp_is_building, instead of category_id. No, I did not set that one, so the error message is factually correct.

Is my understanding correct that:

1. Setting an attribute via obj.set_cp(some_name, value) will actually define a label called f'cp_{some_name}' and it is that cp_-prefixed version we should provide in map_by?
2. map_by='class' is special cased and actually accesses an attribute called category_id?

My extra question would be what is the advantage of obj.set_cp(some_name, value) over setattr(obj, some_name, value)? It seems that both could work?

Thanks for your answers!

[themasterlink]

Hey,

Setting an attribute via obj.set_cp(some_name, value) will actually define a label called f'cp_{some_name}' and it is that cp_-prefixed version we should provide in map_by?

Exactly, all custom properties are stored in the properties entity of a blender object: https://docs.blender.org/manual/en/latest/files/data_blocks.html#files-data-blocks-custom-properties

To make sure they were added by us, we add cp_ before it. This also avoids conflicts with attributes from blender_obj, like name.

map_by='class' is special cased and actually accesses an attribute called category_id?

Correct.

You can use map_by=["name", "cp_is_building", "instances"], here the names of the objects would also be saved, these are not custom properties but real attributes of the blender_obj: https://docs.blender.org/api/current/bpy.types.Object.html#bpy.types.Object

Best, Max

[themasterlink]

I assume this has been resolved.

[themasterlink]

Hey,

Thank you @jatentaki again for raising this issue. We decided because of this to change the API to remove this nonsense with the cp_. Will be in the next version of BlenderProc.

Best, Max