Change in behavior for reinterpreted arrays between Julia 1.10 and Julia 1.11

[jClugstor]

I'm getting an issue where the behavior around reinterpreted arrays is different between Julia 1.10 and Julia 1.11.

Julia 1.11:

Julia Version 1.11.3
Commit d63adeda50d (2025-01-21 19:42 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × AMD Ryzen 7 6800U with Radeon Graphics
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 16 virtual cores)

Pkg> st
  [7da242da] Enzyme v0.13.41
⌅ [f6369f11] ForwardDiff v0.10.38
  [1dea7af3] OrdinaryDiffEq v6.95.1

using Enzyme
using ForwardDiff
using OrdinaryDiffEq
p = rand(3)

function dudt(u, p, t)
    u .* p
end

f = ODEFunction(dudt)

ytmp = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
tmp1 = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
fx_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
x_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))

pf = let f = f
    function (out, u, _p, t)
        out .= f(u, _p, t)
        nothing
    end
end
_tmp6 = pf

dup = Enzyme.Duplicated(p, [0.0, 0.0, 0.0])

Enzyme.autodiff(Enzyme.Reverse, Enzyme.Duplicated(pf, _tmp6),
    Enzyme.Const, Enzyme.Duplicated(fx_dual, x_dual),
    Enzyme.Duplicated(ytmp, tmp1),
    dup, Enzyme.Const(0.0))

gives

ERROR: LLVM error: function failed verification (4)
Stacktrace:
  [1] handle_error(reason::Cstring)
    @ LLVM ~/.julia/packages/LLVM/xTJfF/src/core/context.jl:194
  [2] EnzymeCreatePrimalAndGradient(logic::Enzyme.Logic, todiff::LLVM.Function, retType::Enzyme.API.CDIFFE_TYPE, constant_args::Vector{…}, TA::Enzyme.TypeAnalysis, returnValue::Bool, dretUsed::Bool, mode::Enzyme.API.CDerivativeMode, runtimeActivity::Bool, width::Int64, additionalArg::Ptr{…}, forceAnonymousTape::Bool, typeInfo::Enzyme.FnTypeInfo, uncacheable_args::Vector{…}, augmented::Ptr{…}, atomicAdd::Bool)
    @ Enzyme.API ~/.julia/packages/Enzyme/3VNOP/src/api.jl:269
  [3] enzyme!(job::GPUCompiler.CompilerJob{…}, mod::LLVM.Module, primalf::LLVM.Function, TT::Type, mode::Enzyme.API.CDerivativeMode, width::Int64, parallel::Bool, actualRetType::Type, wrap::Bool, modifiedBetween::NTuple{…} where N, returnPrimal::Bool, expectedTapeType::Type, loweredArgs::Set{…}, boxedArgs::Set{…})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:1754
  [4] codegen(output::Symbol, job::GPUCompiler.CompilerJob{…}; libraries::Bool, deferred_codegen::Bool, optimize::Bool, toplevel::Bool, strip::Bool, validate::Bool, only_entry::Bool, parent_job::Nothing)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:4664
  [5] codegen
    @ ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:3450 [inlined]
  [6] _thunk(job::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams}, postopt::Bool)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:5528
  [7] _thunk
    @ ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:5528 [inlined]
  [8] cached_compilation
    @ ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:5580 [inlined]
  [9] thunkbase(mi::Core.MethodInstance, World::UInt64, FA::Type{…}, A::Type{…}, TT::Type, Mode::Enzyme.API.CDerivativeMode, width::Int64, ModifiedBetween::NTuple{…} where N, ReturnPrimal::Bool, ShadowInit::Bool, ABI::Type, ErrIfFuncWritten::Bool, RuntimeActivity::Bool, edges::Vector{…})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:5691
 [10] thunk_generator(world::UInt64, source::LineNumberNode, FA::Type, A::Type, TT::Type, Mode::Enzyme.API.CDerivativeMode, Width::Int64, ModifiedBetween::NTuple{…} where N, ReturnPrimal::Bool, ShadowInit::Bool, ABI::Type, ErrIfFuncWritten::Bool, RuntimeActivity::Bool, self::Any, fakeworld::Any, fa::Type, a::Type, tt::Type, mode::Type, width::Type, modifiedbetween::Type, returnprimal::Type, shadowinit::Type, abi::Type, erriffuncwritten::Type, runtimeactivity::Type)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/3VNOP/src/compiler.jl:5876
 [11] autodiff(::ReverseMode{…}, ::Duplicated{…}, ::Type{…}, ::Duplicated{…}, ::Duplicated{…}, ::Duplicated{…}, ::Const{…})
    @ Enzyme ~/.julia/packages/Enzyme/3VNOP/src/Enzyme.jl:485
 [12] top-level scope
    @ REPL[72]:1
Some type information was truncated. Use `show(err)` to see complete types.

If you do this (note the upper case Nothings):

using Enzyme
using ForwardDiff
using OrdinaryDiffEq
p = rand(3)

function dudt(u, p, t)
    u .* p
end

f = ODEFunction(dudt)

ytmp = reinterpret(ForwardDiff.Dual{Nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
tmp1 = reinterpret(ForwardDiff.Dual{Nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
fx_dual = reinterpret(ForwardDiff.Dual{Nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
x_dual = reinterpret(ForwardDiff.Dual{Nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))

pf = let f = f
    function (out, u, _p, t)
        out .= f(u, _p, t)
        nothing
    end
end
_tmp6 = pf

dup = Enzyme.Duplicated(p, [0.0, 0.0, 0.0])

Enzyme.autodiff(Enzyme.Reverse, Enzyme.Duplicated(pf, _tmp6),
    Enzyme.Const, Enzyme.Duplicated(fx_dual, x_dual),
    Enzyme.Duplicated(ytmp, tmp1),
    dup, Enzyme.Const(0.0))

it works fine. The only difference is that ytmp etc. are no longer ReinterpretedArrays. The thing is, the first one works on Julia 1.10:

Julia Version 1.10.9
Commit 5595d20a287 (2025-03-10 12:51 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × AMD Ryzen 7 6800U with Radeon Graphics
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 16 virtual cores)

(OnePointTen) pkg> st
Status `~/Documents/Work/dev/ODETesting/SciMLSensitivity/OnePointTen/Project.toml`
  [7da242da] Enzyme v0.13.41
⌅ [f6369f11] ForwardDiff v0.10.38
  [1dea7af3] OrdinaryDiffEq v6.95.1

You can run this without problems,

using Enzyme
using ForwardDiff
using OrdinaryDiffEq
p = rand(3)

function dudt(u, p, t)
    u .* p
end

f = ODEFunction(dudt)

ytmp = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
tmp1 = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
fx_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
x_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))

pf = let f = f
    function (out, u, _p, t)
        out .= f(u, _p, t)
        nothing
    end
end
_tmp6 = pf

dup = Enzyme.Duplicated(p, [0.0, 0.0, 0.0])

Enzyme.autodiff(Enzyme.Reverse, Enzyme.Duplicated(pf, _tmp6),
    Enzyme.Const, Enzyme.Duplicated(fx_dual, x_dual),
    Enzyme.Duplicated(ytmp, tmp1),
    dup, Enzyme.Const(0.0))

One thing is that this:

using Enzyme
using ForwardDiff

struct FunctionHolder
    f
end

(f::FunctionHolder)(args...) = f.f(args...)

f = FunctionHolder(dudt)

tmp = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
tmp1 = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
fx_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
x_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))

pf = let f = f
    function (out, u, _p, t)
        out .= f(u, _p, t)
        nothing
    end
end
_tmp6 = pf

dup = Enzyme.Duplicated(p, [0.0, 0.0, 0.0])

Enzyme.autodiff(Enzyme.Reverse, Enzyme.Duplicated(pf, _tmp6),
    Enzyme.Const, Enzyme.Duplicated(fx_dual, x_dual),
    Enzyme.Duplicated(ytmp, tmp1),
    dup, Enzyme.Const(0.0))

works perfectly fine on Julia 1.11, even though they should essentially be doing the exact same thing I think.

Another thing is that if you specify SciMLBase.NoSpecialize that gets rid of the error as well:

using Enzyme
using ForwardDiff
using OrdinaryDiffEq
using SciMLBase
p = rand(3)

function dudt(u, p, t)
    u .* p
end

f = ODEFunction{false, SciMLBase.NoSpecialize}(dudt)

ytmp = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
tmp1 = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
fx_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
x_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))

pf = let f = f
    function (out, u, _p, t)
        out .= f.f(u, _p, t)
        nothing
    end
end
_tmp6 = pf

dup = Enzyme.Duplicated(p, [0.0, 0.0, 0.0])

Enzyme.autodiff(Enzyme.Reverse, Enzyme.Duplicated(pf, _tmp6),
    Enzyme.Const, Enzyme.Duplicated(fx_dual, x_dual),
    Enzyme.Duplicated(ytmp, tmp1),
    dup, Enzyme.Const(0.0))

[jClugstor]

Actually, something that's closer to what's actually happening but doesn't involve ODEFunctions is:

function dudt(u, p, t)
    u .* p
end

struct FunctionHolder{F}
    f::F
end

(bigF::FunctionHolder)(args...) = bigF.f(args...)

ff = FunctionHolder{typeof(dudt)}(dudt)

ytmp = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0,(0.0,0.0, 0.0)), 3))
tmp1 = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
fx_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))
x_dual = reinterpret(ForwardDiff.Dual{nothing,Float64,3}, fill(ForwardDiff.Dual{Nothing}(0.0, (0.0, 0.0, 0.0)), 3))

pf = let f = ff
            function (out, u, _p, t)
                out .= f(u, _p, t)
                nothing
            end
        end
_tmp6 = make_zero(pf)

dup = Enzyme.Duplicated(p, [0.0,0.0,0.0])

Enzyme.autodiff(Enzyme.Reverse, Enzyme.Duplicated(pf, _tmp6),
    Enzyme.Const, Enzyme.Duplicated(fx_dual, x_dual),
    Enzyme.Duplicated(ytmp, tmp1),
    dup, Enzyme.Const(0.0))

since by default SciMLBase.FullSpecialization is used. But this doesn't actually give the same error:

ERROR: Type of ghost or constant type Duplicated{var"#37#38"{FunctionHolder{typeof(dudt)}}} is marked as differentiable.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] enzyme!(job::GPUCompiler.CompilerJob{…}, mod::LLVM.Module, primalf::LLVM.Function, TT::Type, mode::Enzyme.API.CDerivativeMode, width::Int64, parallel::Bool, actualRetType::Type, wrap::Bool, modifiedBetween::NTuple{…} where N, returnPrimal::Bool, expectedTapeType::Type, loweredArgs::Set{…}, boxedArgs::Set{…})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/txp7d/src/compiler.jl:1572
  [3] codegen(output::Symbol, job::GPUCompiler.CompilerJob{…}; libraries::Bool, deferred_codegen::Bool, optimize::Bool, toplevel::Bool, strip::Bool, validate::Bool, only_entry::Bool, parent_job::Nothing)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/txp7d/src/compiler.jl:4658
  [4] codegen
    @ ~/.julia/packages/Enzyme/txp7d/src/compiler.jl:3444 [inlined]
  [5] _thunk(job::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams}, postopt::Bool)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/txp7d/src/compiler.jl:5518
  [6] _thunk
    @ ~/.julia/packages/Enzyme/txp7d/src/compiler.jl:5518 [inlined]
  [7] cached_compilation
    @ ~/.julia/packages/Enzyme/txp7d/src/compiler.jl:5570 [inlined]
  [8] thunkbase(mi::Core.MethodInstance, World::UInt64, FA::Type{…}, A::Type{…}, TT::Type, Mode::Enzyme.API.CDerivativeMode, width::Int64, ModifiedBetween::NTuple{…} where N, ReturnPrimal::Bool, ShadowInit::Bool, ABI::Type, ErrIfFuncWritten::Bool, RuntimeActivity::Bool, edges::Vector{…})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/txp7d/src/compiler.jl:5681
  [9] thunk_generator(world::UInt64, source::LineNumberNode, FA::Type, A::Type, TT::Type, Mode::Enzyme.API.CDerivativeMode, Width::Int64, ModifiedBetween::NTuple{…} where N, ReturnPrimal::Bool, ShadowInit::Bool, ABI::Type, ErrIfFuncWritten::Bool, RuntimeActivity::Bool, self::Any, fakeworld::Any, fa::Type, a::Type, tt::Type, mode::Type, width::Type, modifiedbetween::Type, returnprimal::Type, shadowinit::Type, abi::Type, erriffuncwritten::Type, runtimeactivity::Type)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/txp7d/src/compiler.jl:5866
 [10] autodiff(::ReverseMode{…}, ::Duplicated{…}, ::Type{…}, ::Duplicated{…}, ::Duplicated{…}, ::Duplicated{…}, ::Const{…})
    @ Enzyme ~/.julia/packages/Enzyme/txp7d/src/Enzyme.jl:485
 [11] top-level scope
    @ ~/Documents/Work/dev/ODETesting/SciMLSensitivity/stiff_adjoint_MWE.jl:62
Some type information was truncated. Use `show(err)` to see complete types.

[jClugstor]

This one is related to the Core 2 SciMLSensitivity tests, https://github.com/SciML/SciMLSensitivity.jl/actions/runs/14969197208/job/42046022185

[ChrisRackauckas]

We one is deprioritized a bit from the others since we found a viable workaround https://github.com/SciML/SciMLSensitivity.jl/pull/1191

[wsmoses]

 Type of ghost or constant type Duplicated{var"#37#38"{FunctionHolder{typeof(dudt)}}} is marked as differentiable.

this one is a user-side error, you're marking a non-differentiable type as differentiable

[jClugstor]

Sorry, probably shouldn't have conflated those, I just wasn't sure how it was actually different from the other example and I was trying to find a simpler working example.

The actual error is the ERROR: LLVM error: function failed verification (4) from the example at the top.

[wsmoses]

ok cool, well in any case if you can make a more minimal example that has the same error (equally applies for all the other issues), it'll allow us to get these fixed faster

[jClugstor]

Yeah for sure, the example at the very top is the result of many hours of figuring out exactly how SciMLSensitivity calls Enzyme with what types and trying to recreate it. For some reason I can't get the exact same error message without using ODEFunction, so that's about as simple as I can seem to make it unfortunately.

[wsmoses]

perhaps cc @oscardssmith for support

[oscardssmith]

this is probably Enzyme strugling with FunctionWrappersWrappers

[ChrisRackauckas]

No SciMLSensitivity unwraps.

[jClugstor]

Yeah they get unwrapped, none of the functions in the MWE are FunctionWrappers or FunctionWrapperWrappers.

[wsmoses]

@jClugstor is this still erring?

[ChrisRackauckas]

It at least is no longer an issue because we worked around it.

[wsmoses]

closing as stale, @ChrisRackauckas reopen if it recurs