Rendering with render_mode="rgb_array" does not work properly when rendering multiple envs

[ManfredStoiber]

When using multiple environments, a strange behavior occurs during rendering in mode "rgb_array":

1. When calling the .render() function on one environment, it is displayed correctly
2. When calling it on a second environment, it also works as intended
3. When rendering the first env again, the objects have weird shapes (see output below)

Here is a minimum example for reproduction:

import metaworld
import random
import matplotlib.pyplot as plt

mt50 = metaworld.MT50() # Construct the benchmark, sampling tasks

training_envs = []
for name, env_cls in mt50.train_classes.items():
  env = env_cls(render_mode="rgb_array")
  task = random.choice([task for task in mt50.train_tasks
                        if task.env_name == name])
  env.set_task(task)
  env.camera_name="corner3"
  training_envs.append(env)

# 1. works (image 1)
plt.imshow(training_envs[34].render()) # pick-place-v2
plt.show()

# 2. works (image 2)
plt.imshow(training_envs[1].render()) # basketball-v2
plt.show()

# 3. does not work anymore (image 3)
plt.imshow(training_envs[34].render()) # pick-place-v2
plt.show()

Output:

Also the images are upside down when using different cameras than default, but that's also the case when using only one environment.

I noticed that the problem does not occur with every combination of environments, for example "pick-place-v2" and "reach-v2" work together, whereas "pick-place-v2" and "basketball-v2" don't. Maybe because the former combination use the same objects and the latter use different objects?

Thank you very much!

[reginald-mclean]

Hi! Thanks for opening this issue. This is really interesting. It probably has something to do with Mujoco, not Meta-World itself. Let me investigate a little bit. I can reproduce the issue on my end. If it's not a Meta-World/Gymnasium issue you may have to open an issue with Mujoco.

[reginald-mclean]

@ManfredStoiber this issue is because of the OpenGL context that is used to render by Mujoco. When you render in env1 the viewer is created and given the "current context." Rendering in env2 is the same. However when you go back to env1, the viewer is reused from the first render call and isn't explicitly given the current context back. If you look at the issue I opened Farama-Foundation/Gymnasium#736 you can find a suggested fix that you can try. It seems to work on my machine but let me know if it doesn't work for you.

[ManfredStoiber]

It works! Thank you very much for your quick reply and your help. Best regards

[reginald-mclean]

@ManfredStoiber it seems like this is going to be fixed within Meta-World so I am going to re-open the issue so we can track it.

[zhihou7]

Why the rendered image look like wired? fliped?