Flux.jl icon indicating copy to clipboard operation
Flux.jl copied to clipboard

Published 20 hours ago • verified publisher icon FluxML

Add `WeightNorm` layer

Open mcabbott opened this issue 3 years ago • 1 comments

RFC as to whether this looks right, and whether the approach is a good one.

Needs tests.

Edit: earlier attempts are #993 (Dense only) and #1005. I had completely forgotten this, but the approach here is similar to @CarloLucibello's suggestion in https://github.com/FluxML/Flux.jl/pull/1005#issuecomment-582973027. (But I'm not sure why the forward pass is unchanged in the example there.)

Closes #942

Checklist

  • [ ] Tests are added
  • [ ] Entry in NEWS.md
  • [ ] Documentation, if applicable

mcabbott avatar Aug 29 '22 04:08 mcabbott

I like the approach, would just advise to do performance tests as well. Due to compiler optimizations, I guess/hope this doesn't incur too much penalization. Also, would this work with recurrent layers? Perhaps define a new dispatch when it finds a Recur. That was also a big blocker in #1005, but I don't believe it should block merging this

bhvieira avatar Feb 24 '23 10:02 bhvieira