Add support for retrieving CarlaActor's ID from instance segmentation camera

[suniique]

Description

This PR extends a previous PR #4840 to enable more instance segmentation applications. The previous PR only returns each object's internal ID (determined by Unreal Engine; see right figure). This, however, could not be matched with Carla's Actor ID (i.e., CarlaActor::ActorId) from the Python side. Retrieving Actor ID can be particularly interesting to those relying on instance/panoptic segmentation, multi-object tracking and segmentation (MOTS), etc (e.g., Issue #4938).

This PR supports re-tagging the objects in the runtime with a user-specified internal_id or actor_id style.

• With the actor_id style (see left figure), we can associate the images with Carla's Actor ID and get further information, like 3D bounding boxes, vehicle types, and traffic lights' state, based on the Actor ID. The objects that are not managed by CarlaActor (e.g., sky, buildings, plants) will be regarded as ActorID = 0.
• The internal_id style (see right figure) provides a more detailed segmentation for each object in the scene, where the color is coded using Unreal's internal ID. This is the behavior for the current version. For compatibility reasons, internal_id is also the default value in this PR.

The re-tagging can be triggered by Python API like

client = carla.Client('localhost', 2000)
world = client.get_world()

# spawn actors...
world.set_instance_tagging_style("actor_id")
# capturing data from instance segmentation camera like left figure
world.set_instance_tagging_style("internal_id")
# capturing data from instance segmentation camera like right figure

print(world.get_instance_tagging_style())  # print: internal_id

This PR probably resolves #4938 and enriches the relevant documentation for instance segmentation camera.

Where has this been tested?

• Platform(s): Debian 10
• Python version(s): 3.7, 3.8
• Unreal Engine version(s): 4.26

Possible Drawbacks

The re-tagging will have a little one-time overhead. However, this is neglectable compared with rendering time.

---

This change is 

[update-docs[bot]]

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would update our CHANGELOG.md based on your changes.

[PiyushRaikar]

Hello @suniique , thanks for sharing your work.

I am trying to do some testing with your files but not getting the expected results. Instead of getting ActorID like 3, 4, 5,... etc. I am getting 4 digit numbers. The "ActorId = G + 256*B" does corelate to the 4 digit Id that is generated in Tagger.cpp (Line 72) before the G & B values are encoded but I am not able to corelate it to the ID I get from the pythonAPI.

Following image shows the data that I am referring to:

Please let me know what the issue might be or how do I interpret this data.

Test system specification: Platform(s): Windows 10 Carla: 0.9.13 Python version(s): 3.8 Unreal Engine version(s): 4.26

Thanks!!

[nargoo0328]

Hello @suniique , thanks for sharing your work.

I am trying to do some testing with your files but not getting the expected results. Instead of getting ActorID like 3, 4, 5,... etc. I am getting 4 digit numbers. Also, "ActorId = G + 256*B" does not corelate to the 4 digit Id that is generated in Tagger.cpp Line 72, before the G & B values are encoded.

Following image shows the data that I am referring to:

Please let me know what the issue might be or how do I interpret this data.

Test system specification: Platform(s): Windows 10 Carla: 0.9.13 Python version(s): 3.8 Unreal Engine version(s): 4.26

Thanks!!

I'm getting the same result here. The encoded instance mask is: R:10, G:1, B:217 which does not match to the number "27".

OS: Ubuntu 18.04 Python: 3.6.13 UE4: 4.26

[tdegraaff]

We experienced the same issues as @PiyushRaikar and @nargoo0328 and after a bit of debugging, we found that it is essential to call world.set_instance_tagging_style("actor_id") after the actors were already spawned. It seems like that if the setting is set at the beginning of the script, the actors that are spawned afterwards are being assigned the "internal_ids".

I cannot realy help to identify the cause of this in the code, but nevertheless I hope this information helps.

[milmario]

With these changes, static (non-actor) objects are no longer captured by the instance segmentation camera (makes sense, as they aren't actors but belong to the map?) Would it be possible to identify these objects and still give them the internal_id. See as an example the car in front, which is not captured by the instance_segmentation camera, but by the semantic segmentation camera. Or any other how we still can segment-out these objects?

[cscribano]

I agree with @milmario It would be useful, yet potentially confusing, if the id associated to non-actor object would match the id returned by get_environment_objects()

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.