rlpyt
rlpyt copied to clipboard
astooke
Continue training
Hello and thanks for a great repo!
I have a question concerning retaking the training of an agent. Let's say I have an agent (custom DqnAgent) and it is trained on one day (it is my episode). I want to retake the agent and introduce to new environment of the next day. How would I do that? I guess my question is related to https://github.com/astooke/rlpyt/issues/111. Do I need to save anything else apart of params.pkl? Thanks!
Hi, and thanks! :)
It should all be in params.pkl, which will have the agent state dict, inside of which is the model state dict.
(To save that, set the logger snapshot mode to "last", which can be done with the argument to logger_context(): https://github.com/astooke/rlpyt/blob/85d4e018a919118c6e42fac3e897aa346d84b9c5/rlpyt/utils/logging/context.py#L25
The issue you referenced is about saving the replay buffer, which can also be done very easily (I've been doing that recently). Let us know if you have any questions about that!
Thanks for your answer!
I was proceeding exactly as you mentioned but I still have a question:
what about epsilon? If I just load agent with params.pkl and start training (which should be regarded as continue really) epsilon is set to 1. Should not it retake from the value where it was stopped?
Thanks again!
Ohh, good point! The epsilon might not be loaded...an oversight on my part. You could save epsilon to the agent's state dict and load that back up. Or just when you resume training, manually enter the epsilon you want to use. Might need some custom solution to continue annealing if you stopped in the middle of that. Sorry that isn't already set up.