Atari replay buffer won't fit into memory even with LazyFrames

[p-christ]

Hi,

Does anyone have any suggestions for the best thing to do when your replay buffer for Atari games won't fit into memory? Is there a good way of dealing with this?

I'm already using the LazyFrames which helps a lot. I can't decrease the replay buffer size because i'm trying to replicate a paper and I also can't get a computer with more memory.

[DanielTakeshi]

@p-christ It would be helpful to know what memory constraints you are facing. How many GB of RAM do you have on your machine? Have you at least tried to see how much RAM a smaller replay buffer would use?

[p-christ]

I have 56GB of RAM and am trying to maintain a replay buffer of 1M atari observations (which are lazyframes pointing to numpy arrays of size 84x84x4). With 56GB I can have a replay buffer of about 0.5M, so quite far off the 1M i need. Any ideas or tips?

[DanielTakeshi]

What command are you running? I think I've run the deepq algorithms here with Atari using the command they provide in the README on 32GB machines and it seems to work fine. (Though I usually use 64GB machines, where it works perfectly but I also have a 32GB machine.)

With your 0.5M replay buffer, run a command that runs DQN (and show it here) and then monitor memory with free -ht to see how much is left.

[AnxietyYoungPoet]

@DanielTakeshi I had the similar problem. The command provided in the README runs with a default buffer_size = 10000, so it work fine. I explicitly set buffer_size = 1e6, and the buffer used about 20G RAM when it contained 0.6M frames. If the LazyFrame works correctly, I think it should take not more than 10G RAM to maintain 1M frames.

[AmazingAng]

The fact is, 1M replay buffer with stacked frames will cost more than 28G Ram: 10^6 * 84 * 84 * 4 bytes ~=28 gigabytes So a 56G Ram machine should fit. Maybe you can check other settings.