Enzyme.jl icon indicating copy to clipboard operation
Enzyme.jl copied to clipboard
verified publisher icon EnzymeAD

Cannot handle unknown binary operator or in LogSumExp

Open BioTurboNick opened this issue 1 year ago • 3 comments

Trying to get more information on this one. If there's anything more I can collect please let me know.

Assertion failed: FT, file /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h, line 1374

[32356] signal (22): SIGABRT
in expression starting at REPL[5]:1
crt_sig_handler at C:/workdir/src\signals-win.c:95
atexit at C:\Windows\System32\msvcrt.dll (unknown line)
atexit at C:\Windows\System32\msvcrt.dll (unknown line)
atexit at C:\Windows\System32\msvcrt.dll (unknown line)
atexit at C:\Windows\System32\msvcrt.dll (unknown line)
atexit at C:\Windows\System32\msvcrt.dll (unknown line)
visitCastInst at /workspace/srcdir/Enzyme/enzyme/Enzyme\AdjointGenerator.h:1374
visit at /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/usr/local/include/llvm/IR\InstVisitor.h:111 [inlined]
CreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme\EnzymeLogic.cpp:4297
recursivelyHandleSubfunction at /workspace/srcdir/Enzyme/enzyme/Enzyme\AdjointGenerator.h:5676
visitCallInst at /workspace/srcdir/Enzyme/enzyme/Enzyme\AdjointGenerator.h:6412
visit at /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/usr/local/include/llvm/IR\InstVisitor.h:111 [inlined]
CreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme\EnzymeLogic.cpp:4297
recursivelyHandleSubfunction at /workspace/srcdir/Enzyme/enzyme/Enzyme\AdjointGenerator.h:5676   
visitCallInst at /workspace/srcdir/Enzyme/enzyme/Enzyme\AdjointGenerator.h:6412
visit at /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/usr/local/include/llvm/IR\InstVisitor.h:111 [inlined]
CreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme\EnzymeLogic.cpp:4297
recursivelyHandleSubfunction at /workspace/srcdir/Enzyme/enzyme/Enzyme\AdjointGenerator.h:5676   
visitCallInst at /workspace/srcdir/Enzyme/enzyme/Enzyme\AdjointGenerator.h:6412
visit at /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/usr/local/include/llvm/IR\InstVisitor.h:111 [inlined]
CreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme\EnzymeLogic.cpp:4297
recursivelyHandleSubfunction at /workspace/srcdir/Enzyme/enzyme/Enzyme\AdjointGenerator.h:5676   
visitCallInst at /workspace/srcdir/Enzyme/enzyme/Enzyme\AdjointGenerator.h:6412
visit at /opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/usr/local/include/llvm/IR\InstVisitor.h:111 [inlined]
CreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme\EnzymeLogic.cpp:4297
EnzymeCreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme\CApi.cpp:631
EnzymeCreatePrimalAndGradient at C:\Users\nicho\.julia\packages\Enzyme\8GSlk\src\api.jl:253
unknown function (ip: 000001b241c298c0)
enzyme! at C:\Users\nicho\.julia\packages\Enzyme\8GSlk\src\compiler.jl:4706
unknown function (ip: 000001b241c27efb)
#codegen#19000 at C:\Users\nicho\.julia\packages\Enzyme\8GSlk\src\compiler.jl:7801
codegen at C:\Users\nicho\.julia\packages\Enzyme\8GSlk\src\compiler.jl:6638 [inlined]
_thunk at C:\Users\nicho\.julia\packages\Enzyme\8GSlk\src\compiler.jl:8909
_thunk at C:\Users\nicho\.julia\packages\Enzyme\8GSlk\src\compiler.jl:8909 [inlined]
cached_compilation at C:\Users\nicho\.julia\packages\Enzyme\8GSlk\src\compiler.jl:8950 [inlined] 
thunkbase at C:\Users\nicho\.julia\packages\Enzyme\8GSlk\src\compiler.jl:9082
unknown function (ip: 000001b21804dc83)
#s2067#19055 at C:\Users\nicho\.julia\packages\Enzyme\8GSlk\src\compiler.jl:9219 [inlined]       
#s2067#19055 at .\none:0
GeneratedFunctionStub at .\boot.jl:602
jl_call_staged at C:/workdir/src\method.c:540
ijl_code_for_staged at C:/workdir/src\method.c:593
get_staged at .\compiler\utilities.jl:123
retrieve_code_info at .\compiler\utilities.jl:135 [inlined]
InferenceState at .\compiler\inferencestate.jl:430
typeinf_edge at .\compiler\typeinfer.jl:920
abstract_call_method at .\compiler\abstractinterpretation.jl:629
abstract_call_gf_by_type at .\compiler\abstractinterpretation.jl:95
abstract_call_known at .\compiler\abstractinterpretation.jl:2087
abstract_call at .\compiler\abstractinterpretation.jl:2169
abstract_call at .\compiler\abstractinterpretation.jl:2162
abstract_call at .\compiler\abstractinterpretation.jl:2354
abstract_eval_call at .\compiler\abstractinterpretation.jl:2370
abstract_eval_statement_expr at .\compiler\abstractinterpretation.jl:2380
abstract_eval_statement at .\compiler\abstractinterpretation.jl:2624
abstract_eval_basic_statement at .\compiler\abstractinterpretation.jl:2889
typeinf_local at .\compiler\abstractinterpretation.jl:3098
typeinf_nocycle at .\compiler\abstractinterpretation.jl:3186
_typeinf at .\compiler\typeinfer.jl:247
typeinf at .\compiler\typeinfer.jl:216
typeinf_ext at .\compiler\typeinfer.jl:1051
typeinf_ext_toplevel at .\compiler\typeinfer.jl:1082
typeinf_ext_toplevel at .\compiler\typeinfer.jl:1078
jfptr_typeinf_ext_toplevel_39006 at C:\Users\nicho\.julia\juliaup\julia-1.10.5+0.x64.w64.mingw32\lib\julia\sys.dll (unknown line)
_jl_invoke at C:/workdir/src\gf.c:2895 [inlined]
ijl_apply_generic at C:/workdir/src\gf.c:3077 [inlined]
jl_apply at C:/workdir/src\julia.h:1982 [inlined]
jl_type_infer at C:/workdir/src\gf.c:394
jl_generate_fptr_impl at C:/workdir/src\jitlayers.cpp:504
jl_compile_method_internal at C:/workdir/src\gf.c:2481
jl_compile_method_internal at C:/workdir/src\gf.c:2372 [inlined]
_jl_invoke at C:/workdir/src\gf.c:2887 [inlined]
ijl_apply_generic at C:/workdir/src\gf.c:3077
autodiff at C:\Users\nicho\.julia\packages\Enzyme\8GSlk\src\Enzyme.jl:512 [inlined]
macro expansion at C:\Users\nicho\Repos\DeepLoco.jl\src\train_new.jl:130 [inlined]
macro expansion at C:\Users\nicho\.julia\packages\ProgressLogging\6KXlp\src\ProgressLogging.jl:328 [inlined]
#train!#26 at C:\Users\nicho\Repos\DeepLoco.jl\src\train_new.jl:126
train! at C:\Users\nicho\Repos\DeepLoco.jl\src\train_new.jl:123
unknown function (ip: 000001b218003b57)
#train_network#25 at C:\Users\nicho\Repos\DeepLoco.jl\src\train_new.jl:44
train_network at C:\Users\nicho\Repos\DeepLoco.jl\src\train_new.jl:17
unknown function (ip: 000001b20327888b)
jl_apply at C:/workdir/src\julia.h:1982 [inlined]
do_call at C:/workdir/src\interpreter.c:126
eval_value at C:/workdir/src\interpreter.c:223
eval_stmt_value at C:/workdir/src\interpreter.c:174 [inlined]
eval_body at C:/workdir/src\interpreter.c:635
jl_interpret_toplevel_thunk at C:/workdir/src\interpreter.c:775
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:934
jl_toplevel_eval_flex at C:/workdir/src\toplevel.c:877
ijl_toplevel_eval at C:/workdir/src\toplevel.c:943 [inlined]
ijl_toplevel_eval_in at C:/workdir/src\toplevel.c:985
eval at .\boot.jl:385 [inlined]
eval at .\Base.jl:88 [inlined]
repleval at c:\Users\nicho\.vscode\extensions\julialang.language-julia-1.124.2\scripts\packages\VSCodeServer\src\repl.jl:229
#112 at c:\Users\nicho\.vscode\extensions\julialang.language-julia-1.124.2\scripts\packages\VSCodeServer\src\repl.jl:192
unknown function (ip: 000001b1c0e3b1db)
with_logstate at .\logging.jl:515
with_logger at .\logging.jl:627 [inlined]
#111 at c:\Users\nicho\.vscode\extensions\julialang.language-julia-1.124.2\scripts\packages\VSCodeServer\src\repl.jl:193
unknown function (ip: 000001b203257c8b)
jl_apply at C:/workdir/src\julia.h:1982 [inlined]
jl_f__call_latest at C:/workdir/src\builtins.c:812
#invokelatest#2 at .\essentials.jl:892 [inlined]
invokelatest at .\essentials.jl:889
unknown function (ip: 000001b1c0e39b5b)
jl_apply at C:/workdir/src\julia.h:1982 [inlined]
do_apply at C:/workdir/src\builtins.c:768
#64 at c:\Users\nicho\.vscode\extensions\julialang.language-julia-1.124.2\scripts\packages\VSCodeServer\src\eval.jl:34
unknown function (ip: 000001b1c0e07944)
jl_apply at C:/workdir/src\julia.h:1982 [inlined]
start_task at C:/workdir/src\task.c:1238
Allocations: 153282837 (Pool: 153075065; Big: 207772); GC: 107

Julia Version 1.10.5
Commit 6f3fdf7b36 (2024-08-27 14:19 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 12 × Snapdragon(R) X 12-core X1E80100 @ 3.40 GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, bdver1)
Threads: 1 default, 0 interactive, 1 GC (on 12 virtual cores)

BioTurboNick avatar Oct 05 '24 02:10 BioTurboNick

can you post the reproducer?

wsmoses avatar Oct 05 '24 02:10 wsmoses 

using Enzyme
using Flux
using StatsFuns

function train_network()
    network = Dense(10 => 10, σ)
    Enzyme.autodiff(
        ReverseWithPrimal,
        _applyloss,
        Active,
        Const(evaluate_and_calc_loss),
        Duplicated(network, make_zero(network)),
        Const(rand(Float32, 10, 4))
    )
end

_applyloss(loss, model, d...) = loss(model, d...)

function evaluate_and_calc_loss(model, images)
    pred = model(images)
    return sum(StatsFuns.logsumexp(pred; dims = 1))
end

After this MWE exercise I realized that StatsFuns.logsumexp (courtesy of LogExpFunctions) is an entirely different function than Flux.logsumexp (courtesy of NNLib). Switching to the latter works, the former produces the above crash. And it only happens with the dims argument specified.

BioTurboNick avatar Oct 06 '24 03:10 BioTurboNick

minor update, repl no longer crashes -- but we have an unknown derivative for an orop

wsmoses avatar Oct 27 '25 06:10 wsmoses