Handle masked reverse mode atomic append

[jeremiedb]

The following minimal function results in a segfault (run on Enzyme's main as of 2023-11-26, Julia v1.10.0-rc1). If removing the @turbo annotation, the autodiff works fine.

using LoopVectorization
using Enzyme

function my_conv_B(x, w)
    y = zero(x)
    @turbo for hi in axes(x, 2)
        for k in axes(x, 1)
            y[k] += w[k] * x[k, hi]
        end
    end
    return y
end
x = rand(Float32, 3, 5);
w = rand(Float32, 3);
y = my_conv_B(x, w);
loss_B(x, w) = sum(my_conv_B(x, w))
dw = zero(w);
loss_B(x, w)
grads = Enzyme.autodiff(Reverse, loss_B, Const(x), Duplicated(w, dw));

Below is the launch command along the resulting error info. One thing I noticed is the reference to /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/Support/Casting.h:578. Is it expected to point to LLVM under what appears like a general system path raher than a Julia specific one? I'm on Ubuntu 23.10 and Julia was installed in a generic way (untar the official binaries under home directory and creation of sym link).

jeremiedb@client-X570S-AORUS-PRO-AX:~/github/ADTests.jl$ julia --project=@. --threads=4 experiments/conv-v3-lv.jl 
julia: /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/Support/Casting.h:578: decltype(auto) llvm::cast(From*) [with To = llvm::IntegerType; From = llvm::Type]: Assertion `isa<To>(Val) && "cast<Ty>() argument of incompatible type!"' failed.

[13573] signal (6.-6): Aborted
in expression starting at /home/jeremiedb/github/ADTests.jl/experiments/conv-v3-lv.jl:44
pthread_kill at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
raise at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7fac7ce267da)
__assert_fail at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
cast<llvm::IntegerType, llvm::Type> at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/Support/Casting.h:578 [inlined]
computeExitLimitFromICmp at /workspace/srcdir/Enzyme/enzyme/Enzyme/MustExitScalarEvolution.cpp:432
computeExitLimitFromCondImpl at /workspace/srcdir/Enzyme/enzyme/Enzyme/MustExitScalarEvolution.cpp:304
computeExitLimitFromCondCached at /workspace/srcdir/Enzyme/enzyme/Enzyme/MustExitScalarEvolution.cpp:159
computeExitLimitFromCond at /workspace/srcdir/Enzyme/enzyme/Enzyme/MustExitScalarEvolution.cpp:52
computeExitLimit at /workspace/srcdir/Enzyme/enzyme/Enzyme/MustExitScalarEvolution.cpp:102
getContext at /workspace/srcdir/Enzyme/enzyme/Enzyme/CacheUtility.cpp:555
forceAugmentedReturns at /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.cpp:8198
CreateAugmentedPrimal at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:2350
recursivelyHandleSubfunction at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:5176
visitCallInst at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:6183
delegateCallInst at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:301 [inlined]
visitCall at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209 [inlined]
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:111 [inlined]
CreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:4238
EnzymeCreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/CApi.cpp:582
EnzymeCreatePrimalAndGradient at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/api.jl:141
unknown function (ip: 0x7fabed3931db)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
enzyme! at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/compiler.jl:3025
unknown function (ip: 0x7fabed38de79)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
#codegen#447 at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/compiler.jl:4649
codegen at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/compiler.jl:4232 [inlined]
_thunk at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/compiler.jl:5201
_thunk at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/compiler.jl:5201 [inlined]
cached_compilation at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/compiler.jl:5235 [inlined]
EnzymeAD/Enzyme.jl#484 at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/compiler.jl:5301
JuliaContext at /home/jeremiedb/.julia/packages/GPUCompiler/U36Ed/src/driver.jl:47
unknown function (ip: 0x7fabed33c7b5)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
#s1040#483 at /home/jeremiedb/.julia/packages/Enzyme/fli3h/src/compiler.jl:5253 [inlined]
#s1040#483 at ./none:0
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
GeneratedFunctionStub at ./boot.jl:600
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
jl_call_staged at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/method.c:540
ijl_code_for_staged at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/method.c:593
get_staged at ./compiler/utilities.jl:123
retrieve_code_info at ./compiler/utilities.jl:135 [inlined]
InferenceState at ./compiler/inferencestate.jl:430
typeinf_edge at ./compiler/typeinfer.jl:920
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2083
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_call at ./compiler/abstractinterpretation.jl:2158
abstract_call at ./compiler/abstractinterpretation.jl:2348
abstract_eval_call at ./compiler/abstractinterpretation.jl:2364
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2374
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2618
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2883
typeinf_local at ./compiler/abstractinterpretation.jl:3092
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3180
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_edge at ./compiler/typeinfer.jl:930
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2083
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_apply at ./compiler/abstractinterpretation.jl:1608
abstract_call_known at ./compiler/abstractinterpretation.jl:2000
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_call at ./compiler/abstractinterpretation.jl:2158
abstract_call at ./compiler/abstractinterpretation.jl:2348
abstract_eval_call at ./compiler/abstractinterpretation.jl:2364
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2374
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2618
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2907
typeinf_local at ./compiler/abstractinterpretation.jl:3092
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3180
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_edge at ./compiler/typeinfer.jl:930
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2083
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_apply at ./compiler/abstractinterpretation.jl:1608
abstract_call_known at ./compiler/abstractinterpretation.jl:2000
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_call at ./compiler/abstractinterpretation.jl:2158
abstract_call at ./compiler/abstractinterpretation.jl:2348
abstract_eval_call at ./compiler/abstractinterpretation.jl:2364
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2374
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2618
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2907
typeinf_local at ./compiler/abstractinterpretation.jl:3092
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3180
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_ext at ./compiler/typeinfer.jl:1051
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1082
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1078
jfptr_typeinf_ext_toplevel_35717.1 at /home/jeremiedb/Downloads/julia-1.10.0-rc1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
jl_apply at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/julia.h:1976 [inlined]
jl_type_infer at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:394
jl_generate_fptr_impl at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/jitlayers.cpp:502
jl_compile_method_internal at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2478 [inlined]
jl_compile_method_internal at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2366
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2884 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
jl_apply at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/julia.h:1976 [inlined]
do_call at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:125
eval_value at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:222
eval_stmt_value at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:173 [inlined]
eval_body at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:616
jl_interpret_toplevel_thunk at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:774
jl_toplevel_eval_flex at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/toplevel.c:877
jl_toplevel_eval_flex at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:383 [inlined]
include_string at ./loading.jl:2070
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
_include at ./loading.jl:2130
include at ./Base.jl:489
jfptr_include_46395.1 at /home/jeremiedb/Downloads/julia-1.10.0-rc1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
exec_options at ./client.jl:318
_start at ./client.jl:552
jfptr__start_82758.1 at /home/jeremiedb/Downloads/julia-1.10.0-rc1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
jl_apply at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/julia.h:1976 [inlined]
true_main at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at julia (unknown line)
unknown function (ip: 0x7fac7ce280cf)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 29085878 (Pool: 29035582; Big: 50296); GC: 38
Aborted (core dumped)

[wsmoses]

Should be fixed by: https://github.com/EnzymeAD/Enzyme/pull/1563

Requires jll bump

[jeremiedb]

Thanks for the quick consideration! Unfortunately, the same error is returned following upgrade to latest version:

[[deps.Enzyme]]
deps = ["CEnum", "EnzymeCore", "Enzyme_jll", "GPUCompiler", "LLVM", "Libdl", "LinearAlgebra", "ObjectFile", "Preferences", "Printf", "Random"]
git-tree-sha1 = "4e9d7571ccae47ee15f9ef40f09827323187c022"
repo-rev = "main"
repo-url = "https://github.com/EnzymeAD/Enzyme.jl.git"
uuid = "7da242da-08ed-463a-9acd-ee780be4f1d9"
version = "0.11.10"

[[deps.EnzymeCore]]
deps = ["Adapt"]
git-tree-sha1 = "2efe862de93cd87f620ad6ac9c9e3f83f1b2841b"
uuid = "f151be2c-9106-41f4-ab19-57ee4f262869"
version = "0.6.4"

[[deps.Enzyme_jll]]
deps = ["Artifacts", "JLLWrappers", "LazyArtifacts", "Libdl", "TOML"]
git-tree-sha1 = "fc50e25508991df186a913d96970113fd04ce5f6"
uuid = "7cc45869-7501-5eee-bdea-0790c847d4ef"
version = "0.0.95+0"

[wsmoses]

A jll is a binary object being imported into Julia. Unfortunately you cannot update the jll with the Julia package manager so you'll have to wait for us to bump it (though I did confirm it fixed your code).

On Tue, Nov 28, 2023 at 7:32 AM jeremiedb @.***> wrote:

Thanks for the quick consideration! Unfortunately, the same error is returned following upgrade to latest version:

[[deps.Enzyme]]deps = ["CEnum", "EnzymeCore", "Enzyme_jll", "GPUCompiler", "LLVM", "Libdl", "LinearAlgebra", "ObjectFile", "Preferences", "Printf", "Random"]git-tree-sha1 = "4e9d7571ccae47ee15f9ef40f09827323187c022"repo-rev = "main"repo-url = "https://github.com/EnzymeAD/Enzyme.jl.git"uuid = "7da242da-08ed-463a-9acd-ee780be4f1d9"version = "0.11.10"

[[deps.EnzymeCore]]deps = ["Adapt"]git-tree-sha1 = "2efe862de93cd87f620ad6ac9c9e3f83f1b2841b"uuid = "f151be2c-9106-41f4-ab19-57ee4f262869"version = "0.6.4"

[[deps.Enzyme_jll]]deps = ["Artifacts", "JLLWrappers", "LazyArtifacts", "Libdl", "TOML"]git-tree-sha1 = "fc50e25508991df186a913d96970113fd04ce5f6"uuid = "7cc45869-7501-5eee-bdea-0790c847d4ef"version = "0.0.95+0"

— Reply to this email directly, view it on GitHub https://github.com/EnzymeAD/Enzyme/issues/1691, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJTUXGDQVM7WTC5G3HR5W3YGWARDAVCNFSM6AAAAAA73NV7QWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRZGE4TSMZYGI . You are receiving this because you commented.Message ID: <EnzymeAD/Enzyme. @.***>

[jeremiedb]

Are you referring to https://github.com/JuliaBinaryWrappers/Enzyme_jll.jl? I noted v0.0.95+0 was released yesterday and I used it in the above test. Should I look forward v0.0.96+0?

[wsmoses]

Correct you will need the next jll once it is available. I tagged a new version of Enzyme with the patch last night, which Yggdrassil will hopefully make available shortly.

On Wed, Nov 29, 2023, 8:20 AM jeremiedb @.***> wrote:

Are you referring to https://github.com/JuliaBinaryWrappers/Enzyme_jll.jl? I noted v0.0.95+0 was released yesterday and I used it in the above test. Should I look forward v0.0.96+0?

— Reply to this email directly, view it on GitHub https://github.com/EnzymeAD/Enzyme/issues/1691, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJTUXD55LAXVG5RPVPSQDTYG3O5HAVCNFSM6AAAAAA73NV7QWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZRGM2DKNJXGM . You are receiving this because you commented.Message ID: <EnzymeAD/Enzyme. @.***>

[wsmoses]

Jll bump is merged

[jeremiedb]

@wsmoses With Enzyme v0.11.11, the above my_conv_B still fails to differentiate, but only if executed in an environment where threads > 1. The following shows the stacktrace, where I think the key hint is the "Assertion `!mask' failed" mentionned at the beginning. Let me know if that looks of a different nature and should be opened in a seperate issue.

jeremiedb@client-X570S-AORUS-PRO-AX:~/github/ADTests.jl$ julia --project=@. --threads=2 experiments/conv-v3-lv.jl 
julia: /workspace/srcdir/Enzyme/enzyme/Enzyme/DiffeGradientUtils.cpp:931: void DiffeGradientUtils::addToInvertedPtrDiffe(llvm::Instruction*, llvm::Value*, llvm::Type*, unsigned int, unsigned int, llvm::Value*, llvm::Value*, llvm::IRBuilder<>&, llvm::MaybeAlign, llvm::Value*): Assertion `!mask' failed.

[45830] signal (6.-6): Aborted
in expression starting at /home/jeremiedb/github/ADTests.jl/experiments/conv-v3-lv.jl:44
pthread_kill at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
raise at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7f514e8267da)
__assert_fail at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
addToInvertedPtrDiffe at /workspace/srcdir/Enzyme/enzyme/Enzyme/DiffeGradientUtils.cpp:931
addToInvertedPtrDiffe at /workspace/srcdir/Enzyme/enzyme/Enzyme/DiffeGradientUtils.cpp:1231
visitLoadLike at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:687
handleAdjointForIntrinsic at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:3666
visitIntrinsicInst at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:3610
visitDbgInfoIntrinsic at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:208 [inlined]
visitDbgVariableIntrinsic at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:206 [inlined]
visitDbgDeclareInst at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:203 [inlined]
delegateCallInst at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:289 [inlined]
visitCall at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209 [inlined]
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:111 [inlined]
CreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:4238
recursivelyHandleSubfunction at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:5664
visitCallInst at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:6286
delegateCallInst at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:301 [inlined]
visitCall at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209 [inlined]
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/Instruction.def:209
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:111 [inlined]
CreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:4238
EnzymeCreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/CApi.cpp:582
EnzymeCreatePrimalAndGradient at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/api.jl:141
unknown function (ip: 0x7f50ce39342b)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
enzyme! at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/compiler.jl:3074
unknown function (ip: 0x7f50ce38e0c9)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
#codegen#448 at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/compiler.jl:4698
codegen at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/compiler.jl:4281 [inlined]
_thunk at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/compiler.jl:5272
_thunk at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/compiler.jl:5272 [inlined]
cached_compilation at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/compiler.jl:5306 [inlined]
EnzymeAD/Enzyme.jl#485 at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/compiler.jl:5372
JuliaContext at /home/jeremiedb/.julia/packages/GPUCompiler/U36Ed/src/driver.jl:47
unknown function (ip: 0x7f50ce33ca05)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
#s1038#484 at /home/jeremiedb/.julia/packages/Enzyme/R8qo4/src/compiler.jl:5324 [inlined]
#s1038#484 at ./none:0
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
GeneratedFunctionStub at ./boot.jl:600
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
jl_call_staged at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/method.c:540
ijl_code_for_staged at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/method.c:593
get_staged at ./compiler/utilities.jl:123
retrieve_code_info at ./compiler/utilities.jl:135 [inlined]
InferenceState at ./compiler/inferencestate.jl:430
typeinf_edge at ./compiler/typeinfer.jl:920
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2083
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_call at ./compiler/abstractinterpretation.jl:2158
abstract_call at ./compiler/abstractinterpretation.jl:2348
abstract_eval_call at ./compiler/abstractinterpretation.jl:2364
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2374
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2618
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2883
typeinf_local at ./compiler/abstractinterpretation.jl:3092
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3180
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_edge at ./compiler/typeinfer.jl:930
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2083
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_apply at ./compiler/abstractinterpretation.jl:1608
abstract_call_known at ./compiler/abstractinterpretation.jl:2000
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_call at ./compiler/abstractinterpretation.jl:2158
abstract_call at ./compiler/abstractinterpretation.jl:2348
abstract_eval_call at ./compiler/abstractinterpretation.jl:2364
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2374
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2618
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2907
typeinf_local at ./compiler/abstractinterpretation.jl:3092
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3180
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_edge at ./compiler/typeinfer.jl:930
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2083
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_apply at ./compiler/abstractinterpretation.jl:1608
abstract_call_known at ./compiler/abstractinterpretation.jl:2000
abstract_call at ./compiler/abstractinterpretation.jl:2165
abstract_call at ./compiler/abstractinterpretation.jl:2158
abstract_call at ./compiler/abstractinterpretation.jl:2348
abstract_eval_call at ./compiler/abstractinterpretation.jl:2364
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2374
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2618
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2907
typeinf_local at ./compiler/abstractinterpretation.jl:3092
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3180
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_ext at ./compiler/typeinfer.jl:1051
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1082
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1078
jfptr_typeinf_ext_toplevel_35717.1 at /home/jeremiedb/Downloads/julia-1.10.0-rc1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
jl_apply at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/julia.h:1976 [inlined]
jl_type_infer at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:394
jl_generate_fptr_impl at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/jitlayers.cpp:502
jl_compile_method_internal at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2478 [inlined]
jl_compile_method_internal at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2366
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2884 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
jl_apply at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/julia.h:1976 [inlined]
do_call at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:125
eval_value at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:222
eval_stmt_value at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:173 [inlined]
eval_body at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:616
jl_interpret_toplevel_thunk at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/interpreter.c:774
jl_toplevel_eval_flex at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/toplevel.c:877
jl_toplevel_eval_flex at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:383 [inlined]
include_string at ./loading.jl:2070
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
_include at ./loading.jl:2130
include at ./Base.jl:489
jfptr_include_46395.1 at /home/jeremiedb/Downloads/julia-1.10.0-rc1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
exec_options at ./client.jl:318
_start at ./client.jl:552
jfptr__start_82758.1 at /home/jeremiedb/Downloads/julia-1.10.0-rc1/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:2892 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/gf.c:3074
jl_apply at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/julia.h:1976 [inlined]
true_main at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/builder-amdci4-5/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at julia (unknown line)
unknown function (ip: 0x7f514e8280cf)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 30464515 (Pool: 30412228; Big: 52287); GC: 42
Aborted (core dumped)

[wsmoses]

Upon the next jll bump this error message will now become:

Unimplemented masked atomic fadd for ptr:  %"ptr.1.i331'ipc_unwrap" = inttoptr i64 %81 to <8 x float>*, !dbg !392 dif:  %681 = load <8 x float>, <8 x float>* %"res.i333'de", align 32, !dbg !302 mask:   %m.i = icmp uge <8 x i32> %v.i, <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>, !dbg !166 orig:   %res.i333 = call <8 x float> @llvm.masked.load.v8f32.p0v8f32(<8 x float>* %ptr.1.i331, i32 noundef 4, <8 x i1> %m.i, <8 x float> noundef zeroinitializer) EnzymeAD/Enzyme.jl#21, !dbg !273, !alias.scope !205


Stacktrace:
 [1] macro expansion
   @ ~/.julia/packages/VectorizationBase/xE5Tx/src/llvm_intrin/memory_addr.jl:997
 [2] __vload
   @ ~/.julia/packages/VectorizationBase/xE5Tx/src/llvm_intrin/memory_addr.jl:997
 [3] _vload
   @ ~/.julia/packages/VectorizationBase/xE5Tx/src/strided_pointers/stridedpointers.jl:105
 [4] macro expansion
   @ ~/.julia/packages/LoopVectorization/7gWfp/src/reconstruct_loopset.jl:1107
 [5] _turbo_!
   @ ~/.julia/packages/LoopVectorization/7gWfp/src/reconstruct_loopset.jl:1107
 [6] my_conv_B
   @ ./REPL[3]:3


Stacktrace:
  [1] throwerr(cstr::Cstring)
    @ Enzyme.Compiler ~/git/Enzyme.jl/src/compiler.jl:1283
  [2] macro expansion
    @ ~/.julia/packages/VectorizationBase/xE5Tx/src/llvm_intrin/memory_addr.jl:997 [inlined]
  [3] __vload
    @ ~/.julia/packages/VectorizationBase/xE5Tx/src/llvm_intrin/memory_addr.jl:997 [inlined]
  [4] _vload
    @ ~/.julia/packages/VectorizationBase/xE5Tx/src/strided_pointers/stridedpointers.jl:105 [inlined]
  [5] macro expansion
    @ ~/.julia/packages/LoopVectorization/7gWfp/src/reconstruct_loopset.jl:1107 [inlined]
  [6] _turbo_!
    @ ~/.julia/packages/LoopVectorization/7gWfp/src/reconstruct_loopset.jl:1107 [inlined]
  [7] my_conv_B
    @ ./REPL[3]:3
  [8] loss_B
    @ ./REPL[7]:1 [inlined]
  [9] diffejulia_loss_B_3362wrap
    @ ./REPL[7]:0
 [10] macro expansion
    @ Enzyme.Compiler ~/git/Enzyme.jl/src/compiler.jl:5374 [inlined]
 [11] enzyme_call
    @ Enzyme.Compiler ~/git/Enzyme.jl/src/compiler.jl:5052 [inlined]
 [12] CombinedAdjointThunk
    @ Enzyme.Compiler ~/git/Enzyme.jl/src/compiler.jl:4994 [inlined]
 [13] autodiff
    @ Enzyme ~/git/Enzyme.jl/src/Enzyme.jl:215 [inlined]
 [14] autodiff
    @ Enzyme ~/git/Enzyme.jl/src/Enzyme.jl:238 [inlined]
 [15] autodiff(::ReverseMode{false, FFIABI}, ::typeof(loss_B), ::Const{Matrix{Float32}}, ::Duplicated{Vector{Float32}})
    @ Enzyme ~/git/Enzyme.jl/src/Enzyme.jl:224
 [16] top-level scope
    @ REPL[10]:1

We should still add this, though now it is a purely Enzyme, not Enzyme.jl issue.

However FYI there are EnzymeRules for NNlib.conv if you'd prefer that to hand rolling your own (or using that custom rule for your conv)