dfdx icon indicating copy to clipboard operation
dfdx copied to clipboard

Published 20 hours ago • verified publisher icon coreylowman

Add Weight Decay to optimizers

Open coreylowman opened this issue 1 year ago • 0 comments

  • [ ] Sgd weight_decay: Option<f32>
  • [ ] RMSprop weight_decay: Option<f32>
  • [ ] Adam/AdamW: weight_decay: Option<AdamDecay>, where enum AdamDecay { Vanilla(f32), AdamW(f32) }.

(maybe use different terms for AdamDecay).

coreylowman avatar Jul 20 '22 14:07 coreylowman