EpisodicLifeEnv ignores resets when it feels like it

[jjfiv]

trafficstars

Like, when you're telling it to reset it considers whether you actually died and decides to ignore you. I think I've debugged this several times now, thinking there's a bug in our game's reset but there isn't.

[anyboby]

That is intended behavior, it is the exact function of the EpisodicLifeEnv wrapper. See also Machado et al. (2018), "Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents" for more info on this.

[jjfiv]

This behavior was quite annoying when trying to run experiments with the environment. Eg, hard to determine max score achieved by an agent across restarts if sometimes it refuses to restart!

Because this is clearly intended behavior for learning we did not file this issue on any model repos, but only on our experimentation tool, as a note to our future selves.