Reduce Habitat memory usage

[apoorvkh]

Hi. I am training an ObjectNav model on an 8 GPU system (with 20 simulators per GPU, as to saturate GPU memory). I am only at 30-50% utilization per CPU core, but am noticing that Habitat uses a very large amount of system memory (e.g. 500 GB and growing during training). I have a lesser amount of RAM (~400 GB), so I have added swap space as a quick fix. But I am seeing a 30-50% performance drop because of this.

I am wondering if is possible to reduce memory consumption while maintaining the number of simulators I have running (I did see issue #630). But I would prefer to have GPU memory as the bottleneck and would be happy to put more CPU resources towards data loading instead. For example, can we unload dataset episodes from memory that we won't be seeing for a while during training?

Please let me know if you have any ideas. Thanks!