Aux loss increases after a couple thousand iterations

[makinyilmaz]

First of all I want to thank you for your precious work !

I have a question about aux loss. I am trying to train a video compression framework where I have a flow compressor and a residual compressor. I initialized optimizer and aux optimizer as you stated under examples directory. I set aux learning rate to 1.e-3 and it first decreases as training continues (I sum flow and residual aux losses). However at some point, it starts to increase. I tried to decrease the aux learning rate to 1.e-5 after enough iterations. Aux loss first decreases but bounces back and starts to increase. Do you have any idea what could cause this observation?

Thanks a lot