Asynchronous Deep ReinFlappyBird

This repository contains an implementation of Asynchronous Advantage Actor-Critic (A3C) that teaches an agent to play Flappy Bird.

Performance

Coming soon!

Technical Details

For my tests, these are the training speeds when using a CPU (Intel Xeon E5620 2.40 GHz) or GPU (NVIDIA GTX1070).

Settings

Here are some of the available flags you can set when you train an agent. For the full list, see a3c.py.

Agent settings

• mode / [train, display, visualize] - Which mode you want to activate when you start a session.
• use_gpu / [True, False] - If you have a/want to use GPU to speed up the training process.
• parallel_agent_size - Number of parallel agents to use during training.
• action_size - Numbers of available actions.
• agent_type / [FF, LSTM] - What type of A3C to train the agent with.

Training and Optimizer settings

The current settings are based on or borrowed from the [implemenentation] (https://github.com/miyosuda/async_deep_reinforce) by @miyosuda. They have not yet been optimized for Flappy Bird but rather used as is for now. Tell me settings that perform better than the current ones!

• max_time_step - 40 000 000 - Maximum training steps.
• initial_alpha_low - -5 - LogUniform low limit for learning rate (represents x in 10^x).
• initial_alpha_high - -3 - LogUniform high limit for learning rate (represents x in 10^x).
• gamma - 0.99 - Discount factor for rewards.
• entropy_beta - 0.01 - Entropy regularization constant.
• grad_norm_clip - 40.0- Gradient norm clipping.
• rmsp_alpha - 0.99 - Decay parameter for RMSProp.
• rmsp_epsilon - 0.1 - Epsilon parameter for RMSProp.
• local_t_max - 5- Repeat step size.

Logging

• log_level - Log level [NONE, FULL]
• average_summary - How many episodes to average summary over.

Display

• display_episodes - Numbers of episodes to display.
• average_summary - How many episodes to average summary over.
• display_log_level - Display log level - NONE prints end summary, MID prints episode summary and FULL prints the π-values, state value and reward for every state. [NONE, MID, FULL]

Getting started

To start a training session with the default parameters, run:

$ python a3c.py

To check your progress and possibly compare different experiments in real time, navigate to your async-deep-flappybird folder and start tensorboard by running:

$ tensorboard --logdir summaries/

Enjoy!

Credit

A3C - The A3C implementation used is a modified version by @miyosuda.

Flappy Bird - The Flappy Bird implementation is based on a version by @yenchenlin with som minor adjustments.

—

2016, Babak Toghiani-Rizi