Enable non-Atari environments for Gym, other custom environments

[saltypeanuts]

Following the tutorial (below, first full tutorial on google searching "custom OpenAI Gym environment tutorial") to create a custom OpenAI Gym environment. The environment works (can be stepped through, reset) and interacted with with other RL libraries (ray[rllib]) and manual RL functions (created through python / sklearn, relatively slow).

Environment setup: https://towardsdatascience.com/creating-a-custom-openai-gym-environment-for-stock-trading-be532be3910e

Does not use atari. Does not require a level select parameter. Does not require rendering.

Registering the custom gym environment with Coach:

my_visualization_parameters = VisualizationParameters(render = True, dump_csv = False, dump_signals_to_csv_every_x_episodes = 0)
my_env = gymenv(env = my_env)

errors out giving:

"__init__() missing 3 required positional arguments: 'level', 'frame_skip', and 'visualization_parameters'

None of these are necessary for the custom OpenAI gym environment we've successfully implemented and are trying to interact with with the Coach RL library.

__init__() missing 3 required positional arguments: 'level', 'frame_skip', and 'visualization_parameters'

Going through adding in 1 parameter at a time:

my_env = gymenv(env = my_env, visualization_parameters = my_visualization_parameters)

It works and allow us to skip a lot of the visual output. Totally fine to require if this is needed for Coach's rendering of algorithm progress and not to actually interact with the OpenAI Gym environment.

Still errors giving us:

__init__() missing 2 required positional arguments: 'level' and 'frame_skip'

Next adding in our frame_skip parameter:

my_env = gymenv(env = my_env, visualization_parameters = my_visualization_parameters, frame_skip = 0)

Because we're not rendering anything in Atari, it's kind of irrelevant what we put in for this parameter. It's fine if we can pass in 0 and it just does nothing as a temporary workaround to get the algorithm training. As expected, it allows us to go through with just 1 positional error:

TypeError: __init__() missing 2 required positional arguments: 'level' and 'frame_skip'

Where a workaround is not possible is under 'level'. We do not need anything from the Atari environment loaded (or any game environment for that matter).

First, passing in null:

my_env = gymenv(env = my_env, visualization_parameters = my_visualization_parameters, frame_skip = 0, level = null)

gives us the error:

Error: Attempted to look up malformed environment ID: b"<module 'null' from '/home/my_user/.local/lib/python3.5/site-packages/null.py'>". (Currently all IDs must be of the form ^(?:[\w:-]+\/)?([\w:.-]+)-v(\d+)$.)

Clearly doesn't like having a null put in. Let's try putting level = 0:

my_env = gymenv(env = my_env, visualization_parameters = my_visualization_parameters, frame_skip = 0, level = 0)

It clearly still does not like this (and the same result is given passing 0 as a string, or other integers, or None).

Error: Attempted to look up malformed environment ID: b'0'. (Currently all IDs must be of the form ^(?:[\w:-]+\/)?([\w:.-]+)-v(\d+)$.)

Let's try passing in a level from the tutorial - 'level='BreakoutDeterministic-v4'' from https://github.com/NervanaSystems/coach/blob/master/tutorials/1.%20Implementing%20an%20Algorithm.ipynb

my_env = gymenv(env = my_env, level = 'BreakoutDeterministic-v4', visualization_parameters = my_visualization_parameters, frame_skip = 0)

Nope, there is a an error:

error: No available video device

There doesn't appear to be a work around for not selecting an Atari level when loading in a custom OpenAI gym environment which doesn't explicitly have and use the Atari setting. I think it would greatly benefit the Coach package to allow custom environments from all packages and not require the use of Atari.