[Forward Split Mode] Free internal caches

[LeilaGhaffari]

Is there a way to build Enzyme without any optimization flags? It would help with debugging application codes (libCEED, in my case).

[wsmoses]

What optimization flags do you mean precisely? (e.g. optimizations that Enzyme does, having the Enzyme compiler itself built without optimization)?

Regardless, yes there are such flags for both (-DCMAKE_BUILD_TYPE=Debug for Enzyme compiler itself without optimization flags, and a flag per optimization within Enzyme will disable that optimization).

[LeilaGhaffari]

I have built Enzyme with -DCMAKE_BUILD_TYPE=Debug but when I tried dropping -O2 from this line, the compilation crashed. Same thing happens with my simple tests.

[wsmoses]

Ah, that's different from what I thought you meant, but I can explain what's happening.

Forward split mode does not yet support freeing its internal caches (and will crash attempting to do so). These internal caches will be created if you have a for loop in your differentiated code that isn't unrolled (e.g. optimized away) and creates a value that must be cached.

When turning optimization on in your example, there are no such internal caches, so it runs fine. However, disabling it leads to the issue.

To work around this, you can pass the enzyme_nofree option and it will not try to free the internal caches (i.e. not have a compile error). If there is an internal cache, you can use a call to split reverse mode (__enzyme_reverse) to free it, or you can leak memory.

The longer term solution here is to enable forward split mode to free its caches (and potentially also add a __enzyme_free_caches function), but that will take some time.

[jedbrown]

If the -O1 at the end of this line is changed to -O2 or above, it builds and runs cleanly.

$ /home/jed/usr/mpich-clang/bin/mpicc -I/home/jed/petsc/mpich-clang/include -I/home/jed/petsc/include -I/home/jed/src/libCEED/include -std=c99 -fPIC -Xclang -load -Xclang /home/jed/src/Enzyme/enzyme/build/Enzyme/ClangEnzyme-13.so -flegacy-pass-manager -c -o build/problems/finite-strain-neo-hookean-initial-ad.o /home/jed/src/libCEED/examples/solids/problems/finite-strain-neo-hookean-initial-ad.c -O1
clang: /home/jed/src/Enzyme/enzyme/Enzyme/GradientUtils.h:1681: void GradientUtils::getReverseBuilder(IRBuilder<> &, bool): Assertion `reverseBlocks.size()' failed.
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: clang -I/home/jed/petsc/mpich-clang/include -I/home/jed/petsc/include -I/home/jed/src/libCEED/include -std=c99 -fPIC -Xclang -load -Xclang /home/jed/src/Enzyme/enzyme/build/Enzyme/ClangEnzyme-13.so -flegacy-pass-manager -c -o build/problems/finite-strain-neo-hookean-initial-ad.o /home/jed/src/libCEED/examples/solids/problems/finite-strain-neo-hookean-initial-ad.c -O1 -I/home/jed/usr/mpich-clang/include
1.      <eof> parser at end of file
2.      Per-module optimization passes
3.      Running pass 'Enzyme Pass' on module '/home/jed/src/libCEED/examples/solids/problems/finite-strain-neo-hookean-initial-ad.c'.
 #0 0x00007f552144f0a7 (/usr/lib/libLLVM-13.so+0xb9e0a7)
 #1 0x00007f552144cb1c llvm::sys::CleanupOnSignal(unsigned long) (/usr/lib/libLLVM-13.so+0xb9bb1c)
 #2 0x00007f5521352819 (/usr/lib/libLLVM-13.so+0xaa1819)
 #3 0x00007f55204c3560 __restore_rt libc_sigaction.c:0:0
 #4 0x00007f552051034c __pthread_kill_implementation pthread_kill.c:0:0
 #5 0x00007f55204c34b8 gsignal (/usr/lib/libc.so.6+0x424b8)
 #6 0x00007f55204ad534 abort (/usr/lib/libc.so.6+0x2c534)
 #7 0x00007f55204ad45c _nl_load_domain.cold loadmsgcat.c:0:0
 #8 0x00007f55204bc116 (/usr/lib/libc.so.6+0x3b116)
 #9 0x00007f551dea9a59 GradientUtils::getReverseBuilder(llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>&, bool) (/home/jed/src/Enzyme/enzyme/build/Enzyme/ClangEnzyme-13.so+0x67ea59)
#10 0x00007f551df39b30 AdjointGenerator<AugmentedReturn const*>::getReverseBuilder(llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>&, bool) (/home/jed/src/Enzyme/enzyme/build/Enzyme/ClangEnzyme-13.so+0x70eb30)
#11 0x00007f551df5f047 AdjointGenerator<AugmentedReturn const*>::visitCallInst(llvm::CallInst&) (/home/jed/src/Enzyme/enzyme/build/Enzyme/ClangEnzyme-13.so+0x734047)
#12 0x00007f551df4bdab llvm::InstVisitor<AdjointGenerator<AugmentedReturn const*>, void>::delegateCallInst(llvm::CallInst&) (/home/jed/src/Enzyme/enzyme/build/Enzyme/ClangEnzyme-13.so+0x720dab)
#13 0x00007f551df3939d llvm::InstVisitor<AdjointGenerator<AugmentedReturn const*>, void>::visitCall(llvm::CallInst&) (/home/jed/src/Enzyme/enzyme/build/Enzyme/ClangEnzyme-13.so+0x70e39d)
#14 0x00007f551df3885a llvm::InstVisitor<AdjointGenerator<AugmentedReturn const*>, void>::visit(llvm::Instruction&) (/home/jed/src/Enzyme/enzyme/build/Enzyme/ClangEnzyme-13.so+0x70d85a)
#15 0x00007f551de5395d llvm::InstVisitor<AdjointGenerator<AugmentedReturn const*>, void>::visit(llvm::Instruction*) (/home/jed/src/Enzyme/enzyme/build/Enzyme/ClangEnzyme-13.so+0x62895d)
#16 0x00007f551de3afd0 EnzymeLogic::CreateForwardDiff(llvm::Function*, DIFFE_TYPE, std::vector<DIFFE_TYPE, std::allocator<DIFFE_TYPE> > const&, TypeAnalysis&, bool, DerivativeMode, bool, unsigned int, llvm::Type*, FnTypeInfo const&, std::map<llvm::Argument*, bool, std::less<llvm::Argument*>, std::allocator<std::pair<llvm::Argument* const, bool> > >, AugmentedReturn const*, bool) (/home/jed/src/Enzyme/enzyme/build/Enzyme/ClangEnzyme-13.so+0x60ffd0)
#17 0x00007f551ddf277c (anonymous namespace)::Enzyme::HandleAutoDiff(llvm::CallInst*, llvm::TargetLibraryInfo&, DerivativeMode, bool) Enzyme.cpp:0:0
#18 0x00007f551ddef061 (anonymous namespace)::Enzyme::lowerEnzymeCalls(llvm::Function&, bool&, std::set<llvm::Function*, std::less<llvm::Function*>, std::allocator<llvm::Function*> >&) Enzyme.cpp:0:0
#19 0x00007f551dde936f (anonymous namespace)::Enzyme::runOnModule(llvm::Module&) Enzyme.cpp:0:0
#20 0x00007f55215b6a8a llvm::legacy::PassManagerImpl::run(llvm::Module&) (/usr/lib/libLLVM-13.so+0xd05a8a)
#21 0x00007f55288835b6 clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream> >) (/usr/lib/libclang-cpp.so.13+0x16bc5b6)
#22 0x00007f5528b51e9b (/usr/lib/libclang-cpp.so.13+0x198ae9b)
#23 0x00007f5529151b10 clang::MultiplexConsumer::HandleTranslationUnit(clang::ASTContext&) (/usr/lib/libclang-cpp.so.13+0x1f8ab10)
#24 0x00007f5527bb519a clang::ParseAST(clang::Sema&, bool, bool) (/usr/lib/libclang-cpp.so.13+0x9ee19a)
#25 0x00007f552913ef79 clang::FrontendAction::Execute() (/usr/lib/libclang-cpp.so.13+0x1f77f79)
#26 0x00007f55290e52cf clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/usr/lib/libclang-cpp.so.13+0x1f1e2cf)
#27 0x00007f552918a5f0 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/usr/lib/libclang-cpp.so.13+0x1fc35f0)
#28 0x00005578aee7f84c cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/usr/bin/clang-13+0x1684c)
#29 0x00005578aee81c2d (/usr/bin/clang-13+0x18c2d)
#30 0x00007f5528e4ea99 (/usr/lib/libclang-cpp.so.13+0x1c87a99)
#31 0x00007f55213528e7 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) (/usr/lib/libLLVM-13.so+0xaa18e7)
#32 0x00007f5528e55030 (/usr/lib/libclang-cpp.so.13+0x1c8e030)
#33 0x00007f5528e6d694 clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&) const (/usr/lib/libclang-cpp.so.13+0x1ca6694)
#34 0x00007f5528e6dac9 clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*> >&) const (/usr/lib/libclang-cpp.so.13+0x1ca6ac9)
#35 0x00007f5528e70a42 clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*> >&) (/usr/lib/libclang-cpp.so.13+0x1ca9a42)
#36 0x00005578aee7761b main (/usr/bin/clang-13+0xe61b)
#37 0x00007f55204ae310 __libc_start_call_main libc-start.c:0:0
#38 0x00007f55204ae3c1 __libc_start_main@GLIBC_2.2.5 (/usr/lib/libc.so.6+0x2d3c1)
#39 0x00005578aee78485 _start (/usr/bin/clang-13+0xf485)
clang-13: error: clang frontend command failed with exit code 134 (use -v to see invocation)
clang version 13.0.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
clang-13: note: diagnostic msg:
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-13: note: diagnostic msg: /tmp/finite-strain-neo-hookean-initial-ad-652d44.c
clang-13: note: diagnostic msg: /tmp/finite-strain-neo-hookean-initial-ad-652d44.sh
clang-13: note: diagnostic msg:

********************

[jedbrown]

Aha, that explanation makes sense. Thanks.