Asy_optimizer

[ypxie]

Hi, thanks for this project. I have a question about the optimizer, Is it enough to simply use the original optimizer provided by pytorch? Before opt.step(). if we simply do sth like:

opt = SGD(shared_para, lr=0.01)
for l_p, s_p in zip(local_para, shared_para):
     l_p.grad.data = s_p.grad.data
opt.step()

will this be enough? Thanks!

[rarilurelo]

@ypxie Your code is applicable to only simple optimization method which doesn't have parameters, like SGD. In the paper, they used RMSProp optimizer. It accumulates square of gradients (parameters). They said Async Optimizer should share accumulations, so I can not use the original RMSProp optimizer.

[ypxie]

@rarilurelo thanks for your reply. Can I simply set the state of pytorch optimizer to shared_memory() and add a lock to protect it. In this way, the copy of the optimizer in different thread will be using the same state memory. Does this sound enough? Thanks!

[rarilurelo]

@ypxie Yes!