arcface_torch: There are periodic, sudden drops in losses

[SuyueLiu]

backbone: mbf loss: arcface dataset: glint (msra + asian celeb) lr = 0.1 wd = 1e-4 fp16=True

as shown in the figure, there are periodic, sudden drops in losses, I am curious why this happens.

[Sunshine066]

Ha ha, I have the same question as you. But my cyclical decline is followed by a small sharp rise followed by a steady decline

[SuyueLiu]

Ha ha, I have the same question as you. But my cyclical decline is followed by a small sharp rise followed by a steady decline

Same as you, did you solve this problem?

[411104983]

The same is true for the loss when I train the VIT model, but the final result does not seem to be worse

[Sunshine066]

The same is true for the loss when I train the VIT model, but the final result does not seem to be worse

Yes, it doesn't make any difference to the final result, but I wonder how it works? I don't have this problem when I train other networks

[Sunshine066]

The loss changes periodically with the epoch. At the beginning of each epoch, the loss drops precipitously and increases slowly in the later training process. I do not understand the reason for this phenomenon.Haha,is there expert know the reason?

[HHCorp]

The loss changes periodically with the epoch. At the beginning of each epoch, the loss drops precipitously and increases slowly in the later training process. I do not understand the reason for this phenomenon.Haha,is there expert know the reason?

#377 This may answer your question.