Snapshot-Ensembles icon indicating copy to clipboard operation
Snapshot-Ensembles copied to clipboard

Update per batch, not per epoch

Open GLambard opened this issue 7 years ago • 0 comments

First of all, thank you very much for sharing your implementation of the Snapshot Ensembling. However, you should not use the callbacks.LearningRateScheduler to update the learning rate.

As mentioned in the original paper under the equation 2, "we update the learning rate at each iteration" (i.e batch) "rather than at every epoch" to improve the convergence of short cycles. But, callbacks.LearningRateScheduler in Keras, with the source code here, updates the learning rate at 'on_epoch_begin'.

After this fix, you should improve further your scores.

GLambard avatar Oct 22 '18 03:10 GLambard