Enzyme.jl
Enzyme.jl copied to clipboard
EnzymeAD
Enzyme fails to differentiate where Zygote succeeds: `jl_get_builtin_fptr`
I ran into an issue optimizing hyper parameters of a Gaussian process with Optim and Enzyme. I'm using AbstractGPs and ParameterHandling in my objective function. I have solved this problem with Zygote so it should be differentiable.
Using AutoZygote() with Optim suggests to either set
autodiff=AutoEnzyme(; mode=Enzyme.set_runtime_activity(Enzyme.Forward))
or
autodiff=AutoEnzyme(; mode=Enzyme.set_runtime_activity(Enzyme.Reverse))
which fail with
ERROR: Enzyme.Compiler.EnzymeNoDerivativeError(Cstring(0x00007f4605d3885c))
Stacktrace:
[1] _similar_for
@ ./array.jl:670 [inlined]
[2] _collect
@ ./array.jl:816
[3] collect_similar
@ ./array.jl:720 [inlined]
[4] map
@ ./abstractarray.jl:3371 [inlined]
[5] value
@ ~/.julia/packages/ParameterHandling/WkHof/src/parameters_base.jl:17 [inlined]
[6] fwddiffe4julia_value_32851wrap
@ ~/.julia/packages/ParameterHandling/WkHof/src/parameters_base.jl:0
[7] macro expansion
@ ~/.julia/packages/Enzyme/Qi7lx/src/compiler.jl:5493 [inlined]
[8] enzyme_call
@ ~/.julia/packages/Enzyme/Qi7lx/src/compiler.jl:5027 [inlined]
[9] ForwardModeThunk
@ ~/.julia/packages/Enzyme/Qi7lx/src/compiler.jl:4918 [inlined]
[10] runtime_generic_fwd(activity::Type{…}, runtimeActivity::Val{…}, strongZero::Val{…}, width::Val{…}, RT::Val{…}, f::typeof(ParameterHandling.value), df::Nothing, df_2::Nothing, df_3::Nothing, df_4::Nothing, primal_1::Vector{…}, shadow_1_1::Vector{…}, shadow_1_2::Vector{…}, shadow_1_3::Vector{…}, shadow_1_4::Vector{…})
@ Enzyme.Compiler ~/.julia/packages/Enzyme/Qi7lx/src/rules/jitrules.jl:327
[11] f
@ ~/enzyme-issue/main.jl:25 [inlined]
[12] fwddiffe4julia_f_17712wrap
@ ~/enzyme-issue/main.jl:0
[13] macro expansion
@ ~/.julia/packages/Enzyme/Qi7lx/src/compiler.jl:5493 [inlined]
[14] enzyme_call
@ ~/.julia/packages/Enzyme/Qi7lx/src/compiler.jl:5027 [inlined]
[15] ForwardModeThunk
@ ~/.julia/packages/Enzyme/Qi7lx/src/compiler.jl:4918 [inlined]
[16] autodiff
@ ~/.julia/packages/Enzyme/Qi7lx/src/Enzyme.jl:669 [inlined]
[17] autodiff
@ ~/.julia/packages/Enzyme/Qi7lx/src/Enzyme.jl:538 [inlined]
[18] macro expansion
@ ~/.julia/packages/Enzyme/Qi7lx/src/sugar.jl:726 [inlined]
[19] #gradient#112
@ ~/.julia/packages/Enzyme/Qi7lx/src/sugar.jl:582 [inlined]
[20] value_and_gradient
@ ~/.julia/packages/DifferentiationInterface/alBlj/ext/DifferentiationInterfaceEnzymeExt/forward_onearg.jl:186 [inlined]
[21] value_and_gradient!(::typeof(f), ::Vector{…}, ::DifferentiationInterfaceEnzymeExt.EnzymeForwardGradientPrep{…}, ::AutoEnzyme{…}, ::Vector{…})
@ DifferentiationInterfaceEnzymeExt ~/.julia/packages/DifferentiationInterface/alBlj/ext/DifferentiationInterfaceEnzymeExt/forward_onearg.jl:213
[22] (::NLSolversBase.var"#fg!#14"{…})(_g::Vector{…}, _x::Vector{…})
@ NLSolversBase ~/.julia/packages/NLSolversBase/n7XXO/src/objective_types/oncedifferentiable.jl:53
[23] value_gradient!!(obj::OnceDifferentiable{Float64, Vector{Float64}, Vector{Float64}}, x::Vector{Float64})
@ NLSolversBase ~/.julia/packages/NLSolversBase/n7XXO/src/interface.jl:82
[24] initial_state(method::LBFGS{…}, options::Optim.Options{…}, d::OnceDifferentiable{…}, initial_x::Vector{…})
@ Optim ~/.julia/packages/Optim/7krni/src/multivariate/solvers/first_order/l_bfgs.jl:168
[25] optimize
@ ~/.julia/packages/Optim/7krni/src/multivariate/optimize/optimize.jl:43 [inlined]
[26] #optimize#77
@ ~/.julia/packages/Optim/7krni/src/multivariate/optimize/interface.jl:274 [inlined]
[27] optimize
@ ~/.julia/packages/Optim/7krni/src/multivariate/optimize/interface.jl:264 [inlined]
[28] top-level scope
@ ~/enzyme-issue/main.jl:42
Executing again in the same REPL session changes the error message to
ERROR:
No augmented forward pass found for jl_get_builtin_fptr
at context: %47 = call {} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)* @jl_get_builtin_fptr({} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 140689697642928 to {}*) to {} addrspace(10)*)) #19, !dbg !95
I've setup a small repository to reproduce the error here: https://github.com/FriesischScott/enzyme-issue
Thanks! I'll try to cook up a minimum working example that doesn't call DifferentiationInterface, to see if the error persists
Would also be good to figure out why the error printing is messed up ERROR: Enzyme.Compiler.EnzymeNoDerivativeError(Cstring(0x00007f4605d3885c))
Here's an MWE without DI or Optim:
using Enzyme
using ParameterHandling
using AbstractGPs
using StatsBase
x = [
-5.0 -3.5714285714285716 -2.1428571428571432 -0.7142857142857144 0.7142857142857135 2.1428571428571432 3.571428571428571 5.0 -5.0 -3.5714285714285716 -2.1428571428571432 -0.7142857142857144 0.7142857142857135 2.1428571428571432 3.571428571428571 5.0 -5.0 -3.5714285714285716 -2.1428571428571432 -0.7142857142857144 0.7142857142857135 2.1428571428571432 3.571428571428571 5.0 -5.0 -3.5714285714285716 -2.1428571428571432 -0.7142857142857144 0.7142857142857135 2.1428571428571432 3.571428571428571 5.0 -5.0 -3.5714285714285716 -2.1428571428571432 -0.7142857142857144 0.7142857142857135 2.1428571428571432 3.571428571428571 5.0 -5.0 -3.5714285714285716 -2.1428571428571432 -0.7142857142857144 0.7142857142857135 2.1428571428571432 3.571428571428571 5.0 -5.0 -3.5714285714285716 -2.1428571428571432 -0.7142857142857144 0.7142857142857135 2.1428571428571432 3.571428571428571 5.0 -5.0 -3.5714285714285716 -2.1428571428571432 -0.7142857142857144 0.7142857142857135 2.1428571428571432 3.571428571428571 5.0
-5.0 -5.0 -5.0 -5.0 -5.0 -5.0 -5.0 -5.0 -3.5714285714285716 -3.5714285714285716 -3.5714285714285716 -3.5714285714285716 -3.5714285714285716 -3.5714285714285716 -3.5714285714285716 -3.5714285714285716 -2.1428571428571432 -2.1428571428571432 -2.1428571428571432 -2.1428571428571432 -2.1428571428571432 -2.1428571428571432 -2.1428571428571432 -2.1428571428571432 -0.7142857142857144 -0.7142857142857144 -0.7142857142857144 -0.7142857142857144 -0.7142857142857144 -0.7142857142857144 -0.7142857142857144 -0.7142857142857144 0.7142857142857135 0.7142857142857135 0.7142857142857135 0.7142857142857135 0.7142857142857135 0.7142857142857135 0.7142857142857135 0.7142857142857135 2.1428571428571432 2.1428571428571432 2.1428571428571432 2.1428571428571432 2.1428571428571432 2.1428571428571432 2.1428571428571432 2.1428571428571432 3.571428571428571 3.571428571428571 3.571428571428571 3.571428571428571 3.571428571428571 3.571428571428571 3.571428571428571 3.571428571428571 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
]
y = [
250.0
218.7130362349021
381.5951686797168
538.7296959600167
590.1582673885882
535.880882965431
475.8558933777592
610.0
109.32528113286128
8.06747188671387
112.64056643065388
223.1278633902541
239.57101207830073
161.97001249479385
90.28321532694711
224.42732194918779
195.47271970012494
35.90587255310287
93.83173677634318
169.33361099541855
162.4531445231154
73.19033735943356
1.5035401915868438
147.3094543940025
308.52561432736354
102.31153685964182
125.25197834235732
177.43023740108288
158.88796334860479
69.62515618492293
9.60016659725114
178.72969596001664
348.52561432736366
107.32611411911704
106.94294044148275
147.45939192003337
128.91711786755522
51.316118284048315
14.614743856726365
218.7296959600167
315.4727197001249
50.94960433152854
38.904623073719264
79.42107455226987
72.54060807996669
18.26322365680965
16.547271970012485
267.3094543940025
309.3252811328612
33.14035818408996
21.0953769262807
73.27363598500622
89.7167846730529
70.42482299042062
115.35610162432312
424.4273219491878
530.0
253.8150770512286
253.43190337359434
328.9337775926697
380.36234902124124
407.71761765930864
510.95793419408574
890.0
]
transform = fit(ZScoreTransform, y; dims=1)
y = StatsBase.transform(transform, y)
n = ParameterHandling.positive(1e-5, exp, 1e-6)
t = ParameterHandling.positive([1.0, 1.0])
θ₀, unflatten = ParameterHandling.flatten([t, n])
θ₀ = [0.0, θ₀...]
function f(θ)
θ_u = ParameterHandling.value(unflatten(θ[2:end]))
kernel = SqExponentialKernel() ∘ ARDTransform(θ_u[1])
gp = GP(θ[1], kernel)
return -logpdf(gp(x, θ_u[2]), y)[1]
end
Enzyme.gradient(Enzyme.set_runtime_activity(Enzyme.Reverse), f, θ₀)
julia> Enzyme.gradient(Enzyme.set_runtime_activity(Enzyme.Reverse), f, θ₀)
ERROR:
No augmented forward pass found for jl_get_builtin_fptr
at context: %47 = call {} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)* @jl_get_builtin_fptr({} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 5119221792 to {}*) to {} addrspace(10)*)) #19, !dbg !95
Stacktrace:
[1] _similar_for
@ ./array.jl:670
[2] _collect
@ ./array.jl:816
Stacktrace:
[1] _similar_for
@ ./array.jl:670 [inlined]
A similar error occurs in forward-over-forward:
julia> using DifferentiationInterface
julia> import Enzyme
julia> hvp(
sum,
SecondOrder(AutoEnzyme(; mode=Enzyme.Forward), AutoEnzyme(; mode=Enzyme.Forward)),
[1.0],
([1.0],),
)
ERROR:
No forward mode derivative found for jl_get_builtin_fptr
at context: %10 = call {} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)* @jl_get_builtin_fptr({} addrspace(10)* noundef addrspacecast ({}* inttoptr (i64 4725148592 to {}*) to {} addrspace(10)*)) #14, !dbg !38
Stacktrace:
[1] make_context_shadows
@ ~/Documents/GitHub/Julia/DifferentiationInterface.jl/DifferentiationInterface/ext/DifferentiationInterfaceEnzymeExt/utils.jl:99
[2] prepare_gradient_nokwarg
@ ~/Documents/GitHub/Julia/DifferentiationInterface.jl/DifferentiationInterface/ext/DifferentiationInterfaceEnzymeExt/forward_onearg.jl:151
Stacktrace:
[1] make_context_shadows
@ ~/Documents/GitHub/Julia/DifferentiationInterface.jl/DifferentiationInterface/ext/DifferentiationInterfaceEnzymeExt/utils.jl:99 [inlined]
[2] prepare_gradient_nokwarg
@ ~/Documents/GitHub/Julia/DifferentiationInterface.jl/DifferentiationInterface/ext/DifferentiationInterfaceEnzymeExt/forward_onearg.jl:151
[3] gradient
@ ~/Documents/GitHub/Julia/DifferentiationInterface.jl/DifferentiationInterface/src/first_order/gradient.jl:62
[4] shuffled_gradient
@ ~/Documents/GitHub/Julia/DifferentiationInterface.jl/DifferentiationInterface/src/first_order/gradient.jl:160 [inlined]
[5] fwddiffejulia_shuffled_gradient_360996wrap
@ ~/Documents/GitHub/Julia/DifferentiationInterface.jl/DifferentiationInterface/src/first_order/gradient.jl:0
[6] macro expansion
@ ~/.julia/packages/Enzyme/HWLY7/src/compiler.jl:5717 [inlined]
[7] enzyme_call
@ ~/.julia/packages/Enzyme/HWLY7/src/compiler.jl:5251 [inlined]
[8] ForwardModeThunk
@ ~/.julia/packages/Enzyme/HWLY7/src/compiler.jl:5153 [inlined]
[9] autodiff
@ ~/.julia/packages/Enzyme/HWLY7/src/Enzyme.jl:669 [inlined]
[10] autodiff
@ ~/.julia/packages/Enzyme/HWLY7/src/Enzyme.jl:558 [inlined]
[11] autodiff
@ ~/.julia/packages/Enzyme/HWLY7/src/Enzyme.jl:530 [inlined]
[12] pushforward
@ ~/Documents/GitHub/Julia/DifferentiationInterface.jl/DifferentiationInterface/ext/DifferentiationInterfaceEnzymeExt/forward_onearg.jl:76 [inlined]
[13] hvp
@ ~/Documents/GitHub/Julia/DifferentiationInterface.jl/DifferentiationInterface/src/second_order/hvp.jl:331 [inlined]
[14] hvp(::typeof(sum), ::SecondOrder{AutoEnzyme{…}, AutoEnzyme{…}}, ::Vector{Float64}, ::Tuple{Vector{…}})
@ DifferentiationInterface ~/Documents/GitHub/Julia/DifferentiationInterface.jl/DifferentiationInterface/src/second_order/hvp.jl:75
[15] top-level scope
@ ~/Documents/GitHub/Julia/DifferentiationInterface.jl/DifferentiationInterface/test/Back/Enzyme/playground.jl:37
Some type information was truncated. Use `show(err)` to see complete types.
can this be isolated to just calls to autodiff (and as isolated to Enzyme as possible)?
This one is a bit trickier to extract because the Enzyme error happens in the DI machinery surrounding context management, not in differentiation per se. I just put it here so that there would be a trace, I'll try to boil it down some day soon
understood, but this error specifically seems to come from julia compiler specific choice of type instabilities. So having the exact code to reproduce it is helpful (and also narrowing down may help DI + Enzyme be faster and/or better supported in general)