functional-transformer icon indicating copy to clipboard operation
functional-transformer copied to clipboard

Published 20 hours ago • verified publisher icon awf

Bug? Loss curves show a distinct correlation with batch id.

Open awf opened this issue 2 years ago • 0 comments

WandB loss curves (e.g. here) show a sawtooth form, correlated with batch ID.

Batches are randomized and this occurs even with 1-bit gradients, so it's not Adam...

image

awf avatar May 27 '22 13:05 awf