Mohamed Tarek

The problem is that most AD packages don't actually support mutation.

You can "import" rules into ReverseDiff using `@grad_from_chainrules`. But the macro doesn't support callable structs and it needs this PR (https://github.com/JuliaDiff/ReverseDiff.jl/pull/232) to support custom input types. But we will need...

We define an internal function which takes the implicit function as its first argument. Then we define the rule on the internal function. ```julia (f::ImplicitFunction)(args...; kwargs...) = call_implicit_function(f, args...; kwargs...)...

preferably we should move all rules to be consistent, defining that function is just an extra line of code

I think some of the tests assumed the previous behavior which is why they are failing. Disclaimer: I said on slack I am not sure this "fix" is even correct,...

We would need a method for `rrule_via_ad` and the `ReverseDiffRuleConfig` first, similar to https://github.com/FluxML/Zygote.jl/blob/master/src/compiler/chainrules.jl#L248. Then we can change that line you pointed to.

Use the `ReverseDiff.@grad` macro to define an rrule for any function that has a branch. The rrule can use `AbstractDifferentiation` to call ReverseDiff again. This will essentially maintain a sub-tape...

It would not be compiled by default but you can choose to compile 2 different tapes, one for each branch. I think you might also be able to do that...

It's not easy. If you are already familiar with ReverseDiff, try reading https://github.com/JuliaDiff/AbstractDifferentiation.jl/blob/master/ext/AbstractDifferentiationReverseDiffExt.jl to understand the AD API. Then you will need to address https://github.com/JuliaDiff/AbstractDifferentiation.jl/issues/41. Then it should be easy...

>One question from me: would this help with being able to represent dynamically-bounded loops on the tape without requiring recompilation? I can think of a few cases related to sequence/time...