Enzyme.jl Lots of "function cannot be proven read-only" failures suddenly popping up

  Function argument passed to autodiff cannot be proven readonly.
  If the the function argument cannot contain derivative data, instead call autodiff(Mode, Const(f), ...)
  See https://enzyme.mit.edu/index.fcgi/julia/stable/faq/#Activity-of-temporary-storage for more information.
  The potentially writing call is   store {} addrspace(10)* %.fca.0.1.extract, {} addrspace(10)** %.fca.0.1.gep, align 8, !dbg !9, !noalias !21, using   %.fca.0.1.gep = getelementptr inbounds { { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, double }, [1 x i8] }, { { {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, {} addrspace(10)*, double }, [1 x i8] }* %.innerparm, i64 0, i32 0, i32 1, !dbg !9

https://github.com/SciML/OrdinaryDiffEq.jl/actions/runs/14539831121/job/40795481724?pr=2677#step:6:1384

https://github.com/SciML/OrdinaryDiffEq.jl/actions/runs/14539831121/job/40795503066?pr=2677#step:6:1392

Seen in multiple CIs. Any clue as to where this is coming from? Was there a recent breaking change in Enzyme?

Apr 19 '25 11:04 ChrisRackauckas

There hasn't been a release of Enzyme.jl in a while, so unlikely?

Do you have an MWE?

Apr 19 '25 13:04 vchuravy

@gdalle since the error is coming from DI

Apr 19 '25 17:04 wsmoses

@ChrisRackauckas did you ever use to run these tests with Enzyme being called directly through its native API? Or was Enzyme not tested at all in OrdinaryDiffEq before the DI overhaul? Most importantly, was there ever a point in time where these tests passed with the DI version? If the answer to the last question is yes, and these tests only started failing today, then the culprit may be v0.18.36 of Enzyme (released today). So I wouldn't be so sure that the error is "coming from" DI, because the Enzyme bindings there haven't changed in a while. I presume something was altered in Enzyme regarding the way that functions are proven to be read-only?

DI offers a solution for this, which is to use AutoEnzyme(function_annotation=Enzyme.Const). Now that my PR on exception types has been merged here, once Enzyme tags a new release, DI can use these types to provide nicer error hints for situations like the current one. But it doesn't change the fact that if these failures appeared today, Enzyme's behavior possibly changed for real, and DI may just be the messenger.

Apr 19 '25 18:04 gdalle

@gdalle That patch release basically only contained your error type PR. So if only that patch errors we can revert and re patch

Apr 19 '25 18:04 wsmoses

@ChrisRackauckas can you try your stuff with the previous version of Enzyme?

Apr 19 '25 18:04 gdalle

In the interim I reverted your error PR and re patched, since that is the only thing that touched anything vaguely related.

Once this is fixed @gdalle feel free to re open your PR (ideally with a test if it is the cause here)

Apr 19 '25 18:04 wsmoses

These tests were passing until yesterday. I can try re-running the tests and see what happens with the new patch.

Apr 19 '25 19:04 ChrisRackauckas

Seems like something is still sad. @gdalle can you get a reproducer without DI?

Apr 19 '25 21:04 wsmoses

Honestly this stuff is buried deep inside OrdinaryDiffEq, so the complexity doesn't come from DI itself. People used to the SciML stack will get it done much faster (also I'm on vacation this week). The truth is that, as more and more packages build upon DI, it will get harder and harder to boil everything down to a pure-Enzyme MWE. In this case, DI enables using Enzyme inside an ODE, so yes we see failures that we may not see outside of ODEs. Whether that's a good or a bad thing depends on your goals, but I'd say it's rather a good thing.

There were four PRs in the Enzyme v0.18.36 release, which seems to have caused the breakage. We've established it wasn't my error PR, which leaves only three. You don't have any clue which of those it could be and why? Or could it be a change in the jll instead of the Julia bindings? Or another dep?

Apr 20 '25 07:04 gdalle

The three PRs in v0.13.36 are:

#2350 (Fairly straight-forward since it just filters out some pre-header instructions)
#2352 (Show to be unrelated (and I would have been very surprised if that was the case, since it was essentiatlly NFC))
#2342 Likely unrelated since it fixes an error during compilation

v0.13.35:

Supports GPUCompiler 1.3

@ChrisRackauckas do you know which Enzyme version was used prior? It might be the case that there was an incompatibility somewhere else that blocked a couple of updates?

Apr 20 '25 16:04 vchuravy

Yeah there similarly hasn't been a jll release for like a month (I just made one, after all this).

Maybe the GPUCompiler compat meant you had an old Enzyme?

Apr 20 '25 16:04 wsmoses

If it helps, here's a CI run from 2 weeks ago:

https://github.com/SciML/OrdinaryDiffEq.jl/actions/runs/14367632037/job/40284221662

Looks like we had a lull for a week in that repo. The one from 2 weeks ago was fine, the Aqua tests fail for a different reason (just an unfinished PR branch), the Taylor one didn't run, and then the initdt one from 4 days ago

https://github.com/SciML/OrdinaryDiffEq.jl/actions/runs/14522100726/job/40745257902#step:6:1388

seems to be the first occurrence of it in our CI. So a good week in there, longer time frame than I'd thought, but maybe that helps narrow down package version changes?

The failing one was:

[a0c0ee7d] DifferentiationInterface v0.6.52
[7da242da] Enzyme v0.13.35
[f151be2c] EnzymeCore v0.8.8
[7cc45869] Enzyme_jll v0.0.173+

while the one that passed was:

[a0c0ee7d] DifferentiationInterface v0.6.50
[f151be2c] EnzymeCore v0.8.8
[7da242da] Enzyme v0.13.35
[7cc45869] Enzyme_jll v0.0.173+0

So my guess then is that it must be DI related?

Apr 21 '25 18:04 ChrisRackauckas

Testing this theory here: https://github.com/SciML/OrdinaryDiffEq.jl/pull/2680

Apr 21 '25 18:04 ChrisRackauckas

Nothing weird is jumping at me https://github.com/JuliaDiff/DifferentiationInterface.jl/compare/DifferentiationInterface-v0.6.50...DifferentiationInterface-v0.6.52

Apr 21 '25 19:04 vchuravy

Nothing between these two DI versions seems to have anything to do with Enzyme. Can the change of behavior be linked to some other part of the SciML stack, which would have made Enzyme's read-only analysis behave differently?

Apr 21 '25 21:04 gdalle

Update is that setting DI to v0.6.50 didn't work. https://github.com/SciML/OrdinaryDiffEq.jl/actions/runs/14579019896/job/40891421732?pr=2680#step:6:1381 . I have no idea what it could be then. I don't even know what I'm looking for. Did a Julia patch come out that I don't know about or something?

Apr 22 '25 00:04 ChrisRackauckas

Can you provide an MWE with two complete manifests, one in which it fails and one in which it succeeds? It's easier than digging through CI. Given the minor differences in DI and Enzyme though, I suspect this is a SciML issue

Apr 22 '25 07:04 gdalle

Okay, I figured out how to do it and what's kind of going on. using Enzyme in a different test triggers the ability to use Enzyme for vjps in the line search method of NonlinearSolve.jl. So a different change enabled this to now test more things.

But... it's not clear why this would fail in the first place?

using FastClosures, NonlinearSolve, Enzyme

t = 0.0
p = [0.04, 3e7, 1e4]
u₀ = [1.0, 0, 0]
du₀ = [-0.04, 0.04, 0.0]
tspan = (0.0, 100000.0)
differential_vars = [true, true, false]

function f(du, u, p, t)
    [-p[1] * u[1] + p[3] * u[2] * u[3] - du[1],
        +p[1] * u[1] - p[2] * u[2]^2 - p[3] * u[2] * u[3] - du[2],
        u[1] + u[2] + u[3] - 1.0]
end
prob_oop = DAEProblem{false}(f, du₀, u₀, tspan, p, differential_vars = differential_vars)

u = [1.0, 0, 0]
du = [-0.04, 0.04, 0.0]
nlequation = @closure (x, _) -> begin
    du_ = ifelse.(differential_vars, x, du)
    u_ = ifelse.(differential_vars, u, x)
    prob_oop.f(du_, u_, prob_oop.p, t)
end

nlfunc = NonlinearFunction(nlequation)
nlprob = NonlinearProblem(nlfunc, ifelse.(differential_vars, du, u))

nlsolve = FastShortcutNonlinearPolyalg()
nlsol = solve(nlprob, nlsolve)

When I pull this out of the package context, it works just fine. It seems like the pass for "Function argument passed to autodiff cannot be proven readonly." is module dependent? Running the same function in OrdinaryDiffEqNonlinearSolve causes it to not pass the test for Enzyme, but then in my REPL it's fine?

It seems like not a desired property for the test to pass/fail based on some randomness like that, so it seems like there is still something to do there.