dopamine
dopamine copied to clipboard
google
Poor introduction to dopamine
So I tried following the overview and - to be honest - this has been a pain.
The entry point to the standard Atari 2600 experiment is [dopamine/discrete_domains/train.py](https://github.com/google/dopamine/blob/master/dopamine/discrete_domains/train.py). To run the basic DQN agent,
python -um dopamine.discrete_domains.train \
--base_dir /tmp/dopamine_runs \
--gin_files dopamine/agents/dqn/configs/dqn.gin
This raises loads of errors:
- First all the missing imports (I fixed those using
pipreqs). - Then ModuleNotFoundError: No module named 'dopamine.metrics' (described in GH-196) that I fixed by changing all
from dopamine.metricstofrom metrics. - Finaly I've got Gym complaining it doesn't distribute ROMs anymore:
gym.error.Error: We're Unable to find the game "Pong". Note: Gym no longer distributes ROMs. If you own a license to use the necessary ROMs for research purposes you can download them via `pip install gym[accept-rom-license]`. Otherwise, you should try importing "Pong" via the command `ale-import-roms`. If you believe this is a mistake perhaps your copy of "Pong" is unsupported. To check if this is the case try providing the environment variable `PYTHONWARNINGS=default::ImportWarning:ale_py.roms`. For more information see: https://github.com/mgbellemare/Arcade-Learning-Environment#rom-management
I believe all of this should run onto the core and atari containers that I created following the readme but I don't see where this is explained...
So I started looking at the colabs hoping to find some more infos but I got loads of errors...
agents.ipynb in Load baseline data
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
[<ipython-input-6-76246b013715>](https://localhost:8080/#) in <module>()
1 # @title Load baseline data
2 get_ipython().system('gsutil -q -m cp -R gs://download-dopamine-rl/preprocessed-benchmarks/* /content/')
----> 3 experimental_data = colab_utils.load_baselines('/content')
1 frames
[/usr/lib/python3.7/copyreg.py](https://localhost:8080/#) in _reconstructor(cls, base, state)
41 def _reconstructor(cls, base, state):
42 if base is object:
---> 43 obj = object.__new__(cls)
44 else:
45 obj = base.__new__(cls, state)
TypeError: object.__new__(BlockManager) is not safe, use BlockManager.__new__()
load_statistics.ipynb in Load the baseline data
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-4-fed9ae80a060> in <module>()
2
3 get_ipython().system('gsutil -q -m cp -R gs://download-dopamine-rl/preprocessed-benchmarks/* /content/')
----> 4 experimental_data = colab_utils.load_baselines('/content')
1 frames
/usr/lib/python3.7/copyreg.py in _reconstructor(cls, base, state)
41 def _reconstructor(cls, base, state):
42 if base is object:
---> 43 obj = object.__new__(cls)
44 else:
45 obj = base.__new__(cls, state)
TypeError: object.__new__(BlockManager) is not safe, use BlockManager.__new__()
agent_visualizer.ipynb in Generate the video
pygame 2.1.2 (SDL 2.0.16, Python 3.7.13)
Hello from the pygame community. https://www.pygame.org/contribute.html
WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
[<ipython-input-3-05aad0365c1e>](https://localhost:8080/#) in <module>()
4 example_viz_lib.run(agent='rainbow', game='SpaceInvaders', num_steps=num_steps,
5 root_dir='/tmp/agent_viz', restore_ckpt='/tmp/tf_ckpt-199',
----> 6 use_legacy_checkpoint=True)
15 frames
[/usr/local/lib/python3.7/dist-packages/atari_py/games.py](https://localhost:8080/#) in get_game_path(game_name)
18 path = os.path.join(_games_dir, game_name) + ".bin"
19 if not os.path.exists(path):
---> 20 raise Exception('ROM is missing for %s, see https://github.com/openai/atari-py#roms for instructions' % (game_name,))
21 return path
22
Exception: ROM is missing for space_invaders, see https://github.com/openai/atari-py#roms for instructions
In call to configurable 'create_atari_environment' (<function create_atari_environment at 0x7f4b9399da70>)
In call to configurable 'Runner' (<class 'dopamine.discrete_domains.run_experiment.Runner'>)
agent_visualizer.ipynb in Generate video
WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
[<ipython-input-5-527301e7cc72>](https://localhost:8080/#) in <module>()
12 gin.parse_config(config)
13 runner = example_viz_lib.MyRunner(base_dir, '/tmp/ckpt.199',
---> 14 AGENT_CREATORS[agent])
15 runner.visualize(base_dir / 'images', num_global_steps=200)
13 frames
[/usr/local/lib/python3.7/dist-packages/atari_py/games.py](https://localhost:8080/#) in get_game_path(game_name)
18 path = os.path.join(_games_dir, game_name) + ".bin"
19 if not os.path.exists(path):
---> 20 raise Exception('ROM is missing for %s, see https://github.com/openai/atari-py#roms for instructions' % (game_name,))
21 return path
22
Exception: ROM is missing for breakout, see https://github.com/openai/atari-py#roms for instructions
In call to configurable 'create_atari_environment' (<function create_atari_environment at 0x7f9a4a7ff200>)
In call to configurable 'Runner' (<class 'dopamine.discrete_domains.run_experiment.Runner'>)
happily tensorboard.ipynb functions properly and cartpole.ipynb too.
Overall, all of this makes for a very poor introduction to dopamine.
sorry about this! we made some changes to the dopamine API recently, which probably broke a few things. i'll look into this next week and have it resolved quickly.
apologies for the inconvenience, but thanks for raising it!
cheers,
-psc
On Thu, May 19, 2022 at 9:51 AM Simon Taeter @.***> wrote:
So I tried following the overview https://google.github.io/dopamine/docs/ and - to be honest - this has been a pain.
The entry point to the standard Atari 2600 experiment is dopamine/discrete_domains/train.py. To run the basic DQN agent,
python -um dopamine.discrete_domains.train
--base_dir /tmp/dopamine_runs
--gin_files dopamine/agents/dqn/configs/dqn.ginThis raises loads of errors; first all the missing imports (I fixed those using pipreqs) then Gym complaining it doesn't distribute ROMs anymore:
gym.error.Error: We're Unable to find the game "Pong". Note: Gym no longer distributes ROMs. If you own a license to use the necessary ROMs for research purposes you can download them via
pip install gym[accept-rom-license]. Otherwise, you should try importing "Pong" via the commandale-import-roms. If you believe this is a mistake perhaps your copy of "Pong" is unsupported. To check if this is the case try providing the environment variablePYTHONWARNINGS=default::ImportWarning:ale_py.roms. For more information see: https://github.com/mgbellemare/Arcade-Learning-Environment#rom-managementI realized all of this should run onto the core and atari containers that I created but I don't see where this is explained.
So I started looking at the colabs hoping to find some more infos but I got loads of errors...
agents.ipynb https://colab.research.google.com/github/google/dopamine/blob/master/dopamine/colab/agents.ipynb in Load baseline data
TypeError Traceback (most recent call last)
in () 1 # @title Load baseline data 2 get_ipython().system('gsutil -q -m cp -R gs://download-dopamine-rl/preprocessed-benchmarks/* /content/') ----> 3 experimental_data = colab_utils.load_baselines('/content') 1 frames
/usr/lib/python3.7/copyreg.py in _reconstructor(cls, base, state) 41 def _reconstructor(cls, base, state): 42 if base is object: ---> 43 obj = object.new(cls) 44 else: 45 obj = base.new(cls, state)
TypeError: object.new(BlockManager) is not safe, use BlockManager.new()
load_statistics.ipynb https://colab.research.google.com/github/google/dopamine/blob/master/dopamine/colab/load_statistics.ipynb in Load the baseline data
TypeError Traceback (most recent call last)
in () 2 3 get_ipython().system('gsutil -q -m cp -R gs://download-dopamine-rl/preprocessed-benchmarks/* /content/') ----> 4 experimental_data = colab_utils.load_baselines('/content') 1 frames
/usr/lib/python3.7/copyreg.py in _reconstructor(cls, base, state) 41 def _reconstructor(cls, base, state): 42 if base is object: ---> 43 obj = object.new(cls) 44 else: 45 obj = base.new(cls, state)
TypeError: object.new(BlockManager) is not safe, use BlockManager.new()
agent_visualizer.ipynb https://colab.research.google.com/github/google/dopamine/blob/master/dopamine/colab/agent_visualizer.ipynb in Generate the video
pygame 2.1.2 (SDL 2.0.16, Python 3.7.13) Hello from the pygame community. https://www.pygame.org/contribute.html WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version. Instructions for updating: non-resource variables are not supported in the long term
Exception Traceback (most recent call last)
in () 4 example_viz_lib.run(agent='rainbow', game='SpaceInvaders', num_steps=num_steps, 5 root_dir='/tmp/agent_viz', restore_ckpt='/tmp/tf_ckpt-199', ----> 6 use_legacy_checkpoint=True) 15 frames
/usr/local/lib/python3.7/dist-packages/atari_py/games.py in get_game_path(game_name) 18 path = os.path.join(_games_dir, game_name) + ".bin" 19 if not os.path.exists(path): ---> 20 raise Exception('ROM is missing for %s, see https://github.com/openai/atari-py#roms for instructions' % (game_name,)) 21 return path 22
Exception: ROM is missing for space_invaders, see https://github.com/openai/atari-py#roms for instructions In call to configurable 'create_atari_environment' (<function create_atari_environment at 0x7f4b9399da70>) In call to configurable 'Runner' (<class 'dopamine.discrete_domains.run_experiment.Runner'>)
agent_visualizer.ipynb https://colab.research.google.com/github/google/dopamine/blob/master/dopamine/colab/jax_agent_visualizer.ipynb in Generate video
WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version. Instructions for updating: non-resource variables are not supported in the long term
Exception Traceback (most recent call last)
in () 12 gin.parse_config(config) 13 runner = example_viz_lib.MyRunner(base_dir, '/tmp/ckpt.199', ---> 14 AGENT_CREATORS[agent]) 15 runner.visualize(base_dir / 'images', num_global_steps=200) 13 frames
/usr/local/lib/python3.7/dist-packages/atari_py/games.py in get_game_path(game_name) 18 path = os.path.join(_games_dir, game_name) + ".bin" 19 if not os.path.exists(path): ---> 20 raise Exception('ROM is missing for %s, see https://github.com/openai/atari-py#roms for instructions' % (game_name,)) 21 return path 22
Exception: ROM is missing for breakout, see https://github.com/openai/atari-py#roms for instructions In call to configurable 'create_atari_environment' (<function create_atari_environment at 0x7f9a4a7ff200>) In call to configurable 'Runner' (<class 'dopamine.discrete_domains.run_experiment.Runner'>)
happily tensorboard.ipynb https://colab.research.google.com/github/google/dopamine/blob/master/dopamine/colab/tensorboard.ipynb functions properly and cartpole.ipynb https://colab.research.google.com/github/google/dopamine/blob/master/dopamine/colab/cartpole.ipynb too.
Overall, all of this makes for a very poor introduction to dopamine.
— Reply to this email directly, view it on GitHub https://github.com/google/dopamine/issues/197, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE3CCMNBOKQ7OCA5RRT3OCLVKZBMJANCNFSM5WMJTZGQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>
hi simon,
the metrics issue should be resolved now, and for the ROMs you should be
able to get that working with
pip install gym[accept-rom-license]
(this is an issue that is separate from us).
i'll look into the colab issues next week, but hopefully this unblocks you for now!
On Thu, May 19, 2022 at 9:51 AM Simon Taeter @.***> wrote:
So I tried following the overview https://google.github.io/dopamine/docs/ and - to be honest - this has been a pain.
The entry point to the standard Atari 2600 experiment is dopamine/discrete_domains/train.py. To run the basic DQN agent,
python -um dopamine.discrete_domains.train
--base_dir /tmp/dopamine_runs
--gin_files dopamine/agents/dqn/configs/dqn.ginThis raises loads of errors; first all the missing imports (I fixed those using pipreqs) then Gym complaining it doesn't distribute ROMs anymore:
gym.error.Error: We're Unable to find the game "Pong". Note: Gym no longer distributes ROMs. If you own a license to use the necessary ROMs for research purposes you can download them via
pip install gym[accept-rom-license]. Otherwise, you should try importing "Pong" via the commandale-import-roms. If you believe this is a mistake perhaps your copy of "Pong" is unsupported. To check if this is the case try providing the environment variablePYTHONWARNINGS=default::ImportWarning:ale_py.roms. For more information see: https://github.com/mgbellemare/Arcade-Learning-Environment#rom-managementI realized all of this should run onto the core and atari containers that I created but I don't see where this is explained.
So I started looking at the colabs hoping to find some more infos but I got loads of errors...
agents.ipynb https://colab.research.google.com/github/google/dopamine/blob/master/dopamine/colab/agents.ipynb in Load baseline data
TypeError Traceback (most recent call last)
in () 1 # @title Load baseline data 2 get_ipython().system('gsutil -q -m cp -R gs://download-dopamine-rl/preprocessed-benchmarks/* /content/') ----> 3 experimental_data = colab_utils.load_baselines('/content') 1 frames
/usr/lib/python3.7/copyreg.py in _reconstructor(cls, base, state) 41 def _reconstructor(cls, base, state): 42 if base is object: ---> 43 obj = object.new(cls) 44 else: 45 obj = base.new(cls, state)
TypeError: object.new(BlockManager) is not safe, use BlockManager.new()
load_statistics.ipynb https://colab.research.google.com/github/google/dopamine/blob/master/dopamine/colab/load_statistics.ipynb in Load the baseline data
TypeError Traceback (most recent call last)
in () 2 3 get_ipython().system('gsutil -q -m cp -R gs://download-dopamine-rl/preprocessed-benchmarks/* /content/') ----> 4 experimental_data = colab_utils.load_baselines('/content') 1 frames
/usr/lib/python3.7/copyreg.py in _reconstructor(cls, base, state) 41 def _reconstructor(cls, base, state): 42 if base is object: ---> 43 obj = object.new(cls) 44 else: 45 obj = base.new(cls, state)
TypeError: object.new(BlockManager) is not safe, use BlockManager.new()
agent_visualizer.ipynb https://colab.research.google.com/github/google/dopamine/blob/master/dopamine/colab/agent_visualizer.ipynb in Generate the video
pygame 2.1.2 (SDL 2.0.16, Python 3.7.13) Hello from the pygame community. https://www.pygame.org/contribute.html WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version. Instructions for updating: non-resource variables are not supported in the long term
Exception Traceback (most recent call last)
in () 4 example_viz_lib.run(agent='rainbow', game='SpaceInvaders', num_steps=num_steps, 5 root_dir='/tmp/agent_viz', restore_ckpt='/tmp/tf_ckpt-199', ----> 6 use_legacy_checkpoint=True) 15 frames
/usr/local/lib/python3.7/dist-packages/atari_py/games.py in get_game_path(game_name) 18 path = os.path.join(_games_dir, game_name) + ".bin" 19 if not os.path.exists(path): ---> 20 raise Exception('ROM is missing for %s, see https://github.com/openai/atari-py#roms for instructions' % (game_name,)) 21 return path 22
Exception: ROM is missing for space_invaders, see https://github.com/openai/atari-py#roms for instructions In call to configurable 'create_atari_environment' (<function create_atari_environment at 0x7f4b9399da70>) In call to configurable 'Runner' (<class 'dopamine.discrete_domains.run_experiment.Runner'>)
agent_visualizer.ipynb https://colab.research.google.com/github/google/dopamine/blob/master/dopamine/colab/jax_agent_visualizer.ipynb in Generate video
WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version. Instructions for updating: non-resource variables are not supported in the long term
Exception Traceback (most recent call last)
in () 12 gin.parse_config(config) 13 runner = example_viz_lib.MyRunner(base_dir, '/tmp/ckpt.199', ---> 14 AGENT_CREATORS[agent]) 15 runner.visualize(base_dir / 'images', num_global_steps=200) 13 frames
/usr/local/lib/python3.7/dist-packages/atari_py/games.py in get_game_path(game_name) 18 path = os.path.join(_games_dir, game_name) + ".bin" 19 if not os.path.exists(path): ---> 20 raise Exception('ROM is missing for %s, see https://github.com/openai/atari-py#roms for instructions' % (game_name,)) 21 return path 22
Exception: ROM is missing for breakout, see https://github.com/openai/atari-py#roms for instructions In call to configurable 'create_atari_environment' (<function create_atari_environment at 0x7f9a4a7ff200>) In call to configurable 'Runner' (<class 'dopamine.discrete_domains.run_experiment.Runner'>)
happily tensorboard.ipynb https://colab.research.google.com/github/google/dopamine/blob/master/dopamine/colab/tensorboard.ipynb functions properly and cartpole.ipynb https://colab.research.google.com/github/google/dopamine/blob/master/dopamine/colab/cartpole.ipynb too.
Overall, all of this makes for a very poor introduction to dopamine.
— Reply to this email directly, view it on GitHub https://github.com/google/dopamine/issues/197, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE3CCMNBOKQ7OCA5RRT3OCLVKZBMJANCNFSM5WMJTZGQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>