Maybe AbstractDifferentiation should shrink to a collection of names?

[gdalle]

I increasingly feel like all the heavy lifting should be done by backends, and AbstractDifferentiation.jl should just define common names.

Whenever we do something that a backend does not support, or when we recode it instead of letting the package do it, we introduce correctness, readability or performance issues. Some examples:

• additional closures with type instabilities (#109, #121)
• hard-to-parse macros (#123)
• vicious tuples (#99, #53)
• no exploitation of optimized backend code (#54)

Moreover, the complexity of the package also makes it very hard to push Enzyme support over the finish line (#84), which is arguably the biggest missing feature.

@adrhill and I are working on a prototype to play with this idea:

https://github.com/gdalle/DifferentiationInterface.jl

It may or may not be merged into AbstractDifferentiation, depending on what the team here thinks. The main caveat I see is that higher order cannot be well supported with an empty shell, but for gradients and JVPs/VJPs it should be close to optimal.

[devmotion]

I agree mostly, I think. Basically, as eg. mentioned in some other issues, my take is that I think AbstractDifferentiation should provide 1) a consistent interface for AD systems and 2) reasonable default implementations for eg jacobian based on pullback etc - but I think each backend (or currently extension) should define the interface explicitly without magic defaults. What I mean is that you should explicitly define whether to use an AbstractDifferentation default for a function or another existing functionality. I assume that backends might want to switch some of their existing functionality to such defaults if they treat AbstractDifferentiation as a first class dependency at some point; and the defaults are useful in early development stage if a package does not want to re-implement jacobian etc functionality. But currently we should try to be as consistent as possible with the AD packages and use their APIs whenever possible.