[Feature Request] Provide documentation on how to use CatFrames with a data collector and replay buffer for images

[AlexandreBrown]

Motivation

Using CatFrames for inference is fairly straightforward and is already well documented.
That being said, using CatFrames to reconstruct a stack of frames when sampling from the replay buffer is not so straightforward I find (subjective) and is not explicitly documentd for images (objective). Using frame stacking for Visual RL is very common practice so I feel like the community would benefit from getting a better documentation on how to use CatFrames for images.

Solution

Provide a clear documentation that explains everything in details (no magic flags/magic values) on how to use CatFrames with a data collector and replay buffer (both extend and sample() method should be shown) for images.

I have created a gist of my attempt to use CatFrames for images and while the inference part works, the stack frames retrieved from the replay buffer do not make sense.

https://gist.github.com/AlexandreBrown/fe378f26a87bdc40c5995dcc7d42f482

Any help on how to make the last part where we sample from the replay buffer return the correct CatFrames data is appreciated.

Contributions

I am willing to work on the PR for the documentation update if someone can help me get the MVP script working.

Checklist

• [x] I have checked that there is no similar issue in the repo (required)