A question regarding the algorithm

[ssydasheng]

Congrats on the awesome work, you really present a useful and yet simple method. However, I still have an question regarding the algorithm :

Since the update is based on the SNR: mu / sigma, thus small SNR can be due to small mu or large variance. If the variance is large, then this parameter has little effect to the predictions of the just-trained-on task, then it can be optimized with large learning rates without affecting the prediction on this task. However, if the mu is small, after optimizing with a new task, the mu might grow large. Then I don't see a reason why it will not affect the predictions on the previous task.

Could you please clarify this for me ? Thanks