Reproducing figure 4 results from Decoupling Representation Learning from RL paper for atari games

[harmanpreet93]

Dear @astooke

Our team is trying to reproduce the decoupling results for Atari games from paper. More specifically, we are trying to reproduce the results from figure 4. To achieve this, we are trying to run rlpyt/rlpyt/ul/experiments/rl_with_ul/scripts/atari/launch/launch_atari_ppo_with_ul_final_1.py.

The script runs just fine. But we have a question about a few of the launch parameters. The launch file prepares results for all 4 curves from Figure 4 as you have dir_names = ["RL", "RL_UL_init", "RL_UL", "UL"]. However, I believe that ATC and RL: ATC-init (frozen encoder weights) require some kind of experience replay buffers whereas RL+ATC (consists of aux loss) and RL should work just fine without any replay buffers. How does this launch file works without having to use any saved replay buffers? Or does this code not produce results for all 4 curves in one shot and require us to run different launch files for reproducing ATC and RL: ATC-init, or maybe we are still missing some crucial point here. Can you please help us understand?