Generate code for Enzyme autodiff for functions with Pointer/array arguments

[Nirhar]

For functions of type double myfunction(double* arr){...}; OR double myfunction(double arr[3]){...};

we generate derivative code of type:

        void d_myfunction(double* arr, clad::array_ref<double> _d_arr){
		double* d_arr = _d_arr.ptr();
		__enzyme_autodiff_myfunction(myfunction, arr, d_arr);
	}

Further on enzyme will handle the function differentiation

This Commit also Downloads and Installs Enzyme in the Clad Directory, if the flag "-DENZYME=On" is passed to cmake during project configuration.

This Commit also contains tests for the above feature in test/Gradient/Enzyme.C

[Nirhar]

@vgvassilev @parth-07 The clad code in this pull request throws the error

"cannot initialize a parameter of type 'double (double *)' with an lvalue of type 'double (double *)'"

during compilation of the below code:

#include "clad/Differentiator/Differentiator.h"
double f(double *arr) { return arr[0] * arr[1]; }
int main() {
    auto f_grad=clad::gradient<clad::opts::use_enzyme>(f);
}

Any idea how to resolve this?

[parth-07]

@vgvassilev @parth-07 The clad code in this pull request throws the error

"cannot initialize a parameter of type 'double (double *)' with an lvalue of type 'double (double *)'"

during compilation of the below code:

#include "clad/Differentiator/Differentiator.h"
double f(double *arr) { return arr[0] * arr[1]; }
int main() {
    auto f_grad=clad::gradient<clad::opts::use_enzyme>(f);
}

Any idea how to resolve this?

Are you able to print generated derived function? If yes, can you please paste that in a comment here?

[Nirhar]

@parth-07 No, I am not able to print the generated function. The same error is thrown when I attempt to print it.

[parth-07]

@parth-07 No, I am not able to print the generated function. The same error is thrown when I attempt to print it.

Have you tried printing the derived function using the flag -fdump-derived-fn?

[parth-07]

"cannot initialize a parameter of type 'double (double *)' with an lvalue of type 'double (double *)'"

The error suggests that you are using an incorrect value category (https://en.cppreference.com/w/cpp/language/value_category) for the function argument of the call to __enzyme_autodiff_. Correctly generating clang::DeclRefExpr node may fix the issue. You can use, Sema::BuildDeclRefExpr or VisitorBase::BuildDeclRef for correctly generating it.

[Nirhar]

@parth-07 No, I am not able to print the generated function. The same error is thrown when I attempt to print it.

Have you tried printing the derived function using the flag -fdump-derived-fn?

Yes I tried that, the generated function does not contain the __enzyme_autodiff function call

[Nirhar]

"cannot initialize a parameter of type 'double (double *)' with an lvalue of type 'double (double *)'"

The error suggests that you are using an incorrect value category (https://en.cppreference.com/w/cpp/language/value_category) for the function argument of the call to __enzyme_autodiff_. Correctly generating clang::DeclRefExpr node may fix the issue. You can use, Sema::BuildDeclRefExpr or VisitorBase::BuildDeclRef for correctly generating it.

Okay let me try this out

[Nirhar]

This code runs on my machine, but here the tests throw the error fatal: could not create work tree dir 'Enzyme': Permission denied. How do I deal with this @vgvassilev @parth-07 ?

[Nirhar]

Now I am getting the below error in the CI tests:

CMake Error: Could not open file for write in copy operation /usr/local/EnzymeAD/tmp/Enzyme-mkdirs.cmake.tmp

I am not getting this error on my machine. How do I resolve this @vgvassilev @parth-07?

[parth-07]

Now I am getting the below error in the CI tests:

CMake Error: Could not open file for write in copy operation /usr/local/EnzymeAD/tmp/Enzyme-mkdirs.cmake.tmp

I am not getting this error on my machine. How do I resolve this @vgvassilev @parth-07?

It seems to be a permission issue. Can you try to install it in a non-root directory? And also, check if we are missing the right permissions somewhere.

[Nirhar]

Now I am getting the below error in the CI tests:

CMake Error: Could not open file for write in copy operation /usr/local/EnzymeAD/tmp/Enzyme-mkdirs.cmake.tmp

I am not getting this error on my machine. How do I resolve this @vgvassilev @parth-07?

It seems to be a permission issue. Can you try to install it in a non-root directory? And also, check if we are missing the right permissions somewhere.

Currently I am installing Enzyme in the "CMAKE_INSTALL_PREFIX". I think in the CI machines this is in some root directory. Where do I install it, is the build_dir a good place?

[parth-07]

Now I am getting the below error in the CI tests:

CMake Error: Could not open file for write in copy operation /usr/local/EnzymeAD/tmp/Enzyme-mkdirs.cmake.tmp

I am not getting this error on my machine. How do I resolve this @vgvassilev @parth-07?

It seems to be a permission issue. Can you try to install it in a non-root directory? And also, check if we are missing the right permissions somewhere.

Currently I am installing Enzyme in the "CMAKE_INSTALL_PREFIX". I think in the CI machines this is in some root directory. Where do I install it, is the build_dir a good place?

You can just specify a non-root directory for the CMAKE_INSTALL_PREFIX option in the CMake configure command.

[parth-07]

Where do I install it, is the build_dir a good place?

Any non-root directory should work for the debugging purpose right now. We would like things to work as expected when the user is installing in root locations as well. Thus the main goal is to find the underlying issue that's causing the CMake configuration to fail.

[Nirhar]

Its mentioned in the Enzyme website that Enzyme has been tested against LLVM 7,8,9,10,11,12. I personally tested it against LLVM 13 and 14 and it works. Maybe the lower versions of LLVM/clang are causing a problem? As I have seen, most failures in the CI are because the Enzyme build has failed.

[parth-07]

Its mentioned in the Enzyme website that Enzyme has been tested against LLVM 7,8,9,10,11,12. I personally tested it against LLVM 13 and 14 and it works. Maybe the lower versions of LLVM/clang are causing a problem? As I have seen, most failures in the CI are because the Enzyme build has failed.

Can you test enzyme integration with Clad when lower versions of LLVM are used in your local system? If you are getting the same/similar errors in the local system as well, then it will be easier to debug.

[Nirhar]

@parth-07 Enzyme fails during build in my System when built with llvm 6, with the same errors as thrown by the CI module, for example osx-clang-runtime6

I noticed a few other CI modules fail due to a flag used during compilation with enzyme(after build), an unrecognised flag in lower versions of llvm. I will correct this.

[parth-07]

@parth-07 Enzyme fails during build in my System when built with llvm 6, with the same errors as thrown by the CI module, for example osx-clang-runtime6

I only meant to test enzyme integration with Clad for LLVM versions that are supported by Enzyme. If I recall correctly tests of LLVM versions <= 10 were failing at that time when I commented. It does not make sense to support Enzyme integration with Clad for LLVM versions that are not supported by Enzyme.

[codecov[bot]]

Codecov Report

Merging #466 (3598e48) into master (34aa8ff) will decrease coverage by 0.16%. The diff coverage is 86.84%.

@@            Coverage Diff             @@
##           master     #466      +/-   ##
==========================================
- Coverage   92.68%   92.51%   -0.17%     
==========================================
  Files          35       37       +2     
  Lines        5439     5490      +51     
==========================================
+ Hits         5041     5079      +38     
- Misses        398      411      +13     

Impacted Files	Coverage Δ
include/clad/Differentiator/ReverseModeVisitor.h	98.70% <ø> (-0.02%)	:arrow_down:
tools/ClangBackendPlugin.h	0.00% <0.00%> (ø)
tools/ClangBackendPlugin.cpp	66.66% <66.66%> (ø)
tools/ClangPlugin.cpp	90.21% <81.81%> (-0.59%)	:arrow_down:
lib/Differentiator/ReverseModeVisitor.cpp	95.92% <97.29%> (+0.03%)	:arrow_up:
lib/Differentiator/CladUtils.cpp	97.26% <0.00%> (-0.37%)	:arrow_down:
lib/Differentiator/VisitorBase.cpp	97.60% <0.00%> (-0.05%)	:arrow_down:
lib/Differentiator/ErrorEstimator.cpp	98.53% <0.00%> (-0.05%)	:arrow_down:
... and 8 more

Impacted Files	Coverage Δ
include/clad/Differentiator/ReverseModeVisitor.h	98.70% <ø> (-0.02%)	:arrow_down:
tools/ClangBackendPlugin.h	0.00% <0.00%> (ø)
tools/ClangBackendPlugin.cpp	66.66% <66.66%> (ø)
tools/ClangPlugin.cpp	90.21% <81.81%> (-0.59%)	:arrow_down:
lib/Differentiator/ReverseModeVisitor.cpp	95.92% <97.29%> (+0.03%)	:arrow_up:
lib/Differentiator/CladUtils.cpp	97.26% <0.00%> (-0.37%)	:arrow_down:
lib/Differentiator/VisitorBase.cpp	97.60% <0.00%> (-0.05%)	:arrow_down:
lib/Differentiator/ErrorEstimator.cpp	98.53% <0.00%> (-0.05%)	:arrow_down:
... and 8 more

[vgvassilev]

Can you rebase your PR, I think we have fixed the bot failures.