[Question] Color randomization as in ManiSkillv2

[alexcbb]

Hello,

I'm currently trying to create an environment with different objects and random colors at each episode. I've searched on how to do that and saw an old documentation of ManiSkillv2 but it seems that it does not work this way now : https://github.com/haosulab/ManiSkill/blob/v0.5.3/examples/tutorials/customize_environments.ipynb (in Domain Randomization part)

Would you have a proper way to do it now ? I've been searching a way to access the visual bodies of the actor but didn't find anything.

Thank in advance !

[StoneT2000]

For object randomization you can check how PickSingleYCB task works. Note that in GPU simulation by default reconfiguration only happens once as it is a bit slow.

For color and texture randomization there is a way to do it, i'll provide some examples like before in some documentation later.

[alexcbb]

For object randomization you can check how PickSingleYCB task works. Note that in GPU simulation by default reconfiguration only happens once as it is a bit slow.

For color and texture randomization there is a way to do it, i'll provide some examples like before in some documentation later.

Yes, I've already have the code for object randomization, my issue was more about the color randomization, I've got a solution currently that is only working on CPU and require deleting the actors. But if a better solution exists that would be perfect.

[StoneT2000]

For color/texture randomization of already built objects you can check out the code for PushT: https://github.com/haosulab/ManiSkill/blob/main/mani_skill/envs/tasks/tabletop/push_t.py#L39-L46

Essentially during environment building/reconfiguration, you can modify at runtime the color of objects. For example if you are building a cube object, you use

builder.add_box_visual(
        half_size=[half_size] * 3,
        material=sapien.render.RenderMaterial(
            base_color=color,
        ),
    )
builder.set_scene_idxs([scene_idx]) # optional if you want to only build it in the scene_idx parallel environment

to add a box visual. You can pick a color here during reconfiguration (modify the _load_scene function of your environment). If you want a different color/texture per parallel environment, then make sure that when using the builder, only build the object in one of the sub-scenes, and build it again with a different texture in other sub-scenes

Note that you must reconfigure your environment or else these color changes will not work / there will be errors that show up. You cannot change textures (easily) after reconfiguration.

With that in mind if you use a ton of parallel envs (say 1024), you can have a different randomization in each of the 1024 envs. And then to increase the coverage/domain randomization, you can reconfigure the environment maybe every 20 environment resets to sample new colors/textures/objects (e.g. a new set of ycb objects, different textures etc.).

If your GPU sim has less environments, you can reconfigure the environment more frequently without having to worry about slow downs caused by reconfiguration (reconfiguration time is approximately proportional with the number of parallel envs)

[alexcbb]

For color/texture randomization of already built objects you can check out the code for PushT: https://github.com/haosulab/ManiSkill/blob/main/mani_skill/envs/tasks/tabletop/push_t.py#L39-L46

Essentially during environment building/reconfiguration, you can modify at runtime the color of objects. For example if you are building a cube object, you use

builder.add_box_visual(
        half_size=[half_size] * 3,
        material=sapien.render.RenderMaterial(
            base_color=color,
        ),
    )
builder.set_scene_idxs([scene_idx]) # optional if you want to only build it in the scene_idx parallel environment

to add a box visual. You can pick a color here during reconfiguration (modify the _load_scene function of your environment). If you want a different color/texture per parallel environment, then make sure that when using the builder, only build the object in one of the sub-scenes, and build it again with a different texture in other sub-scenes

Note that you must reconfigure your environment or else these color changes will not work / there will be errors that show up. You cannot change textures (easily) after reconfiguration.

With that in mind if you use a ton of parallel envs (say 1024), you can have a different randomization in each of the 1024 envs. And then to increase the coverage/domain randomization, you can reconfigure the environment maybe every 20 environment resets to sample new colors/textures/objects (e.g. a new set of ycb objects, different textures etc.).

If your GPU sim has less environments, you can reconfigure the environment more frequently without having to worry about slow downs caused by reconfiguration (reconfiguration time is approximately proportional with the number of parallel envs)

Ok, thank you very much for the answer, I'll try this during the day. Would you like me to update the documentation accordingly as it is done in the one of ManiSkillv2 ?

[StoneT2000]

A PR for that would be great! I would recommend adding this stuff to the custom tasks documentation (just add a section at the bottom of loading actors/articulations)

https://github.com/haosulab/ManiSkill/blob/main/docs/source/user_guide/tutorials/custom_tasks/loading_objects.md

[alexcbb]

Before going forward on this, I want to understand something: I've tried the code which seems to work if I don't activate the set_scene_idxs value. The issue is that I use a RecordEpisoder wrapper that make some weird error on the moment it flushes the trajectories : might it be related to the following comment ==> https://github.com/haosulab/ManiSkill/blob/main/mani_skill/utils/wrappers/record.py#L373 ? If yes, I can maybe find a way to solve this issue first (as it is blocking for me)

[StoneT2000]

Yes at the moment it's a bit complicated for how to save trajectories of GPU simulation when parallel environments have a different number of objects or potentially different geometries. It is disabled on purpose. Appreciate any suggestions for how to improve trajectory saving for diverse parallel simulations.

[alexcbb]

For now it is activated by default, but in my specific use case I don't see how I can possible make use of the state of the env. I looked at the explanations on top of the class, but what would be the specific usage of such a state env ? By just deactivating the saving of the state, I can achieve to get a nice color and geometry randomization over scene by loading in advance 5 instances of each geometry per scene. I then select a certain number of gemotries (between 4 and 8) to display randomly in front of the cameras, I put all the others away from the scene (a bit like the hide_visual function of Actor : https://github.com/haosulab/ManiSkill/blob/main/mani_skill/utils/structs/actor.py#L177), but I don't know whether there is a better solution.

[StoneT2000]

Sorry for delay here. You can update the RecordEpisode wrapper to have an option to disable environment state saving for now. It is useful for replaying trajectories and some RL algorithms at the moment.