Zygote.jl
Zygote.jl copied to clipboard
FluxML
Decide what to do about `pushforward` and co.
We get at least a couple of confused questions about it every month, but AFAICT there isn't much appetite to bring it up to a point of general usefulness. How does a big flashing banner saying "don't use this" sound until if/when someone feels the urge to pick it back up?
We added the forward mode to be used, and it's actually the fastest forward mode on neural networks right now, so I would say if we get questions we should help resolve them and flesh it out.
Are there benchmarks floating out there that demonstrate this? I was under the impression that rule support was not fleshed out enough to support all but the most basic of Flux models.
We can do mlps which are of most importance to small model cases such as SciML. Small - medium sized models are also the ones that are poised to gain most from a fast forward mode, beyond which a reverse mode would start making sense, esp where a lot of arguments are required.
Also see https://github.com/FluxML/Zygote.jl/pull/503#issuecomment-635408758
Thanks for the link, now I'm curious why SciML hasn't picked it up for the diffeq(flux) ecosystem. I'm sure there's a perfectly legitimate reason, but I didn't see any discussions about it outside of https://github.com/SciML/NeuralPDE.jl/issues/150. Perhaps @ChrisRackauckas recalls the background there?
Also, we could definitely use some documentation for this feature if it's going to be sticking around. Currently, there are a grand total of 0 doc comments on https://github.com/FluxML/Zygote.jl/tree/master/src/forward, and hence no docs section on the Zygote site either. I'm sure there are PRs somewhere that address this, but when I checked yesterday everything since #503 has either a) been in limbo for over a year or b) closed outright?
No docs and some things ran into issues really quickly. The conclusion was just, wait for Diffractor.
What is the status of this in 2024? @oxinabox any insights? I just had a question asking whether we should support Zygote's forward mode in DI
I have never heard of anyone using it. I have never used it. It was one of the last things Mike did before leaving AFAIK.
