Update Gymnasium to v1.0.0

[pseudo-rnd-thoughts]

This PR updates SB3 to Gymnasium v1.0, read the release-notes to see all the changes.

Motivation and Context

Gymnasium is the core API used in SB3, therefore would be helpful for both SB3 to use the latest version and that SB3 provides a great testing ground to check for that the Gymnasium release works as intended.

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [x] Breaking change (fix or feature that would cause existing functionality to change)
• [ ] Documentation (update in the documentation)

Checklist

• [x] I've read the CONTRIBUTION guide (required)
• [ ] I have updated the changelog accordingly (required).
• [ ] My change requires a change to the documentation.
• [ ] I have updated the tests accordingly (required for a bug fix or a new feature).
• [ ] I have updated the documentation accordingly.
• [ ] I have opened an associated PR on the SB3-Contrib repository (if necessary)
• [ ] I have opened an associated PR on the RL-Zoo3 repository (if necessary)
• [x] I have reformatted the code using make format (required)
• [x] I have checked the codestyle using make check-codestyle and make lint (required)
• [x] I have ensured make pytest and make type both pass. (required)
• [x] I have checked that the documentation builds using make doc (required)

Note: You can run most of the checks using make commit-checks.

Note: we are using a maximum length of 127 characters per line

[pseudo-rnd-thoughts]

There are only two main issues to resolve + need to rewrite VecVideoRecorder

1. As we have removed Env.__getattr__, then we need to update to use Env.get_wrapper_attr
2. Registration of external environment is not automatical, i.e., atari, therefore a hack was added to fix it

[pseudo-rnd-thoughts]

The CI seems to have failed due to reasons unrelated to the version change

[araffin]

Thanks for the PR =)

As we have removed Env.getattr, then we need to update to use Env.get_wrapper_attr

if possible (and if not too hacky), I would add backward compat changes to handle both gymnasium 0.29 and 1.x.

[pseudo-rnd-thoughts]

Thanks for the PR =)

No worries, all the errors seem expected and no unexpected bugs are found

As we have removed Env.getattr, then we need to update to use Env.get_wrapper_attr

if possible (and if not too hacky), I would add backward compat changes to handle both gymnasium 0.29 and 1.x.

I believe the changes made should be backward compatible. Just updating VecRecordEnv needs to be fully updated / rewritten

[Kallinteris-Andreas]

Assuming the Environment which used gymnasium==0.29 is not broken from the gymansium==1.0 update, it should just work without any additional compatibility work in SB3

[araffin]

Assuming the Environment which used gymnasium==0.29 is not broken from the gymansium==1.0 update, it should just work without any additional compatibility work in SB3

I'm talking about allowing people to use 0.29 with SB3.

I believe the changes made should be backward compatible.

mmh, I would double check the getattr() part, I remember it was warning the user

[pseudo-rnd-thoughts]

mmh, I would double check the getattr() part, I remember it was warning the user

If the code works with 1.0.0a1 then it will work with 0.29 but possibly not the other way around

[pseudo-rnd-thoughts]

@araffin I believe I have fixed all the issues except for updating VecVideoRecorder. I don't know how SB3 works internals, would you be able to get one of your devs to update that?

[araffin]

I don't know how SB3 works internals, would you be able to get one of your devs to update that?

Currently, there is only one active dev (me...), Quentin (@qgallouedec ) is helping me with answering questions and doing code reviews, for the rest, we have to rely on the community.

In the meantime, you could try running tests in SB3 contrib and RL Zoo with this branch, that should unveil other bugs/issues.

[Kallinteris-Andreas]

i tested this: (from https://stable-baselines3.readthedocs.io/en/master/guide/examples.html#record-a-video)

import gymnasium as gym
from stable_baselines3.common.vec_env import VecVideoRecorder, DummyVecEnv

env_id = "CartPole-v1"
video_folder = "logs/videos/"
video_length = 100

vec_env = DummyVecEnv([lambda: gym.make(env_id, render_mode="rgb_array")])

obs = vec_env.reset()

# Record the video starting at the first step
vec_env = VecVideoRecorder(vec_env, video_folder,
                       record_video_trigger=lambda x: x == 0, video_length=video_length,
                       name_prefix=f"random-agent-{env_id}")

vec_env.reset()
for _ in range(video_length + 1):
  action = [vec_env.action_space.sample()]
  obs, _, _, _ = vec_env.step(action)
# Save the video
vec_env.close()

and I get this error

python test.py
Traceback (most recent call last):
  File "/home/intelligence-lab-pc4/Documents/kalli/test_mjc3/test.py", line 17, in <module>
    vec_env.reset()
  File "/home/intelligence-lab-pc4/Documents/kalli/test_mjc3/stable-baselines3/stable_baselines3/common/vec_env/vec_video_recorder.py", line 66, in reset
    self.start_video_recorder()
  File "/home/intelligence-lab-pc4/Documents/kalli/test_mjc3/stable-baselines3/stable_baselines3/common/vec_env/vec_video_recorder.py", line 78, in start_video_recorder
    self.video_recorder.capture_frame()
    ^^^^^^^^^^^^^^^^^^^
  File "/home/intelligence-lab-pc4/Documents/kalli/test_mjc3/stable-baselines3/stable_baselines3/common/vec_env/base_vec_env.py", line 420, in __getattr__
    return self.getattr_recursive(name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/intelligence-lab-pc4/Documents/kalli/test_mjc3/stable-baselines3/stable_baselines3/common/vec_env/base_vec_env.py", line 445, in getattr_recursive
    attr = getattr(self.venv, name)
           ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DummyVecEnv' object has no attribute 'video_recorder'

i am using gymnasium==1.0.0a1, sb3 this PR's latest commit (in march)

[pseudo-rnd-thoughts]

Just updating VecRecordEnv needs to be fully updated / rewritten

@Kallinteris-Andreas Yes, this is the only part of the PR that still needs to be done. I'm tempted to just copy and paste the old video recorder in as a solution. Might try to do this evening

[qgallouedec]

Feel free to ping me if necessary

[pseudo-rnd-thoughts]

@araffin I've updated to alpha 2, however, this removed Dict space using OrderedDict for samples as python 3.7+ dicts are ordered, therefore, there is no need for OrderedDict. However, this causes mypy issues, do you understand the problem better?