rl
rl copied to clipboard
pytorch
[Feature Request] Add linear interpolation to FrameSkipTransform
Motivation
I would suggest adding an option for linear interpolation to the FrameSkipTransform (documentation here). This means, instead of repeating the same action frame_skip number of times for each step in the environment, we could instead keep track of action from the previous step and then interpolate towards the new action over frame_skip steps.
This means that at the first step, the previous action would be executed, at the final step, the new action would be executed, and in between, we interpolate linearly between the two actions.
Solution
Add a variant of the FrameSkipTransform, or otherwise add this as an option in the params of FrameSkipTransform.
Checklist
- [x] I have checked that there is no similar issue in the repo (required)
Some pointers: we could want to make this stateless, if a buffer needs to be carried it could be put in the output tensordict under a private entry name, eg "_action_interp_buffer". We can also make it stateful within the transform. Both are viable options. A good example of the former approach is StepCounter or other stateless transforms like TargetReturn. CatFrames is a good example of stateful transform.
Another datapoint: be careful about reset, which will require some clever masking of the buffer with the reset key. Again have a look at the above mentioned transforms to get a sense of how it should work.