OpenMP offload build fails with undefined `camp::resources::Omp`

[abbotts]

I'm not able to get the current develop (81d6990ddb20719ff1a18f723b3bf767b134658b) or the v0.14.0 tag to build with CCE OpenMP offload due to build time failures that seem to originate from CAMP not respecting -DENABLE_TARGET_OPENMP.

I configure with:

cmake -DENABLE_OPENMP=On -DENABLE_TARGET_OPENMP=On \
-DENABLE_CUDA=Off -DENABLE_CLANG_CUDA=Off -DENABLE_HIP=Off \
-DENABLE_CLANG_HIP=Off -DENABLE_TBB=Off \
-DCMAKE_CXX_COMPILER=CC -DCMAKE_C_COMPILER=cc \
-DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=14 \
-DENABLE_TESTS=On -DENABLE_EXAMPLES=Off  ../

And get failures looking like:

[  2%] Building CXX object test/integration/plugin/CMakeFiles/test-plugin-forall-OpenMP.exe.dir/test-plugin-forall-OpenMP.cpp.o
cd /home/users/sabbott/nightly_crons/raja-hip-gfx908/RAJA/build/test/integration/plugin && /opt/cray/pe/craype/2.7.10.1/bin/CC  -DGTEST_HAS_DEATH_TEST=1 -I/home/users/sabbott/nightly_crons/raja-hip-gfx908/RAJA/test/include -isystem /home/users/sabbott/nightly_crons/raja-hip-gfx908/RAJA/blt/thirdparty_builtin/googletest-master-2020-01-07/googletest/include -I/home/users/sabbott/nightly_crons/raja-hip-gfx908/RAJA/test/integration/plugin/tests -I/home/users/sabbott/nightly_crons/raja-hip-gfx908/RAJA/blt/thirdparty_builtin/googletest-master-2020-01-07/googletest -I/home/users/sabbott/nightly_crons/raja-hip-gfx908/RAJA/include -I/home/users/sabbott/nightly_crons/raja-hip-gfx908/RAJA/build/include -I/home/users/sabbott/nightly_crons/raja-hip-gfx908/RAJA/tpl/camp/include  -Wall -Wextra  -O2 -DNDEBUG -fPIE   -fopenmp -std=c++14 -o CMakeFiles/test-plugin-forall-OpenMP.exe.dir/test-plugin-forall-OpenMP.cpp.o -c /home/users/sabbott/nightly_crons/raja-hip-gfx908/RAJA/build/test/integration/plugin/test-plugin-forall-OpenMP.cpp
In file included from /home/users/sabbott/nightly_crons/raja-hip-gfx908/RAJA/build/test/integration/plugin/test-plugin-forall-OpenMP.cpp:11:
In file included from /home/users/sabbott/nightly_crons/raja-hip-gfx908/RAJA/test/include/RAJA_test-base.hpp:15:
In file included from /home/users/sabbott/nightly_crons/raja-hip-gfx908/RAJA/include/RAJA/RAJA.hpp:44:
In file included from /home/users/sabbott/nightly_crons/raja-hip-gfx908/RAJA/include/RAJA/pattern/forall.hpp:64:
In file included from /home/users/sabbott/nightly_crons/raja-hip-gfx908/RAJA/include/RAJA/policy/MultiPolicy.hpp:31:
/home/users/sabbott/nightly_crons/raja-hip-gfx908/RAJA/include/RAJA/util/resource.hpp:112:35: error: no type named 'Omp' in namespace 'camp::resources'
    using type = camp::resources::Omp;
                 ~~~~~~~~~~~~~~~~~^
/home/users/sabbott/nightly_crons/raja-hip-gfx908/RAJA/include/RAJA/util/resource.hpp:117:35: error: no type named 'Omp' in namespace 'camp::resources'
    using type = camp::resources::Omp;
                 ~~~~~~~~~~~~~~~~~^

I ran a bisect and the first broken commit was afa2caf49fbdf25359320c6609dee1a1e47912f4 , but since that was reverted when CAMP was updated it is probably a regression in CAMP. Just quickly looking at CAMP my initial guess is that it's missing a cmake define in the config file or something. I'll try to identify the actual error in camp and open an issue or PR there. Another, less likely, option might be the -DENABLE_TARGET_OPENMP=On option not propagating to CAMP.

[rchen20]

Hi @abbotts, we've made some updates in the Camp repo which should solve this problem (https://github.com/LLNL/camp). If you have a chance, would you mind trying camp/main to see if this fixes your issue? Thanks!

[rhornung67]

I think Tom said the fix was in camp develop??

[rchen20]

Tom did say develop, but he actually meant "main". 🙂

[abbotts]

This got lost in a flood of emails. I'm trying to test now with current RAJA/develop and camp/main, but a stray -ltarget_openmp has made it onto the link line for some reason and I can't progress far enough to build the test.

I'll need to track that down and figure out how to get rid of it before I can test the CAMP changes. I may not have time for that until next week.

[rhornung67]

@abbotts is this (and the following 'foreach' logic) the root of the issue you are seeing? https://github.com/LLNL/camp/blob/main/CMakeLists.txt#L52

@trws thoughts?

[abbotts]

@rhornung67 I think you're right. The else branch is passing target_openmp through to camp_depends:

    else()
      list (APPEND camp_depends ${backend})
    endif()

Which makes it to the blt_add_library here: https://github.com/LLNL/camp/blob/80ff45683238be2c3b78bbc3a2cd7c85605eb2d9/CMakeLists.txt#L107

Then through to the RAJA build after configure via:

tpl/camp/CMakeFiles/Export/lib/cmake/camp/campTargets.cmake:  INTERFACE_LINK_LIBRARIES "target_openmp;RAJA::openmp"

[trws]

That's interesting, it's the right target for camp to be using, and should behave with BLT downstream, but I'm not sure what we should do for non-BLT consumers. They do need to include OpenMP offload support for some of the headers to work, but since that target is BLT specific... Will have to chat this over I think and make sure camp and RAJA match up on the behavior.

[davidbeckingsale]

BLT doesn't provide that target - we handle OpenMP offload by depending on openmp and explicitly putting the offload flags in OpenMP_CXX_FLAGS (e.g. https://github.com/LLNL/RAJA/blob/develop/scripts/lc-builds/blueos_clang_omptarget.sh#L42)

[trws]

Now we get to the stage of debugging where I wonder why it ever worked. 🤦 Ok, that clearly needs to be fixed, I'll add it to the active fix I'm working on, expect it today.

[davidbeckingsale]

@trws did you get a chance to look at this?

[trws]

I think this ends up being addressed with the exported target fixes we were working on, should probably check if RAJA develop/camp main has this working, it should.

[abbotts]

** EDIT ** stand by.. forgot to update submodules so probably still have the old CAMP. Retesting it right now.

No luck with RAJA develop on an internal HPE system using the configure command I opened the ticket with. Still getting:

Scanning dependencies of target test-plugin-workgroup-Sequential.exe
[  3%] Building CXX object test/integration/plugin/CMakeFiles/test-plugin-workgroup-Sequential.exe.dir/test-plugin-workgroup-Sequential.cpp.o
In file included from /home/users/sabbott/git/RAJA/cce_build/test/integration/plugin/test-plugin-workgroup-Sequential.cpp:11:
In file included from /home/users/sabbott/git/RAJA/test/include/RAJA_test-base.hpp:15:
In file included from /home/users/sabbott/git/RAJA/include/RAJA/RAJA.hpp:44:
In file included from /home/users/sabbott/git/RAJA/include/RAJA/pattern/forall.hpp:64:
In file included from /home/users/sabbott/git/RAJA/include/RAJA/policy/MultiPolicy.hpp:31:
/home/users/sabbott/git/RAJA/include/RAJA/util/resource.hpp:119:35: error: no type named 'Omp' in namespace 'camp::resources'
    using type = camp::resources::Omp;
                 ~~~~~~~~~~~~~~~~~^
/home/users/sabbott/git/RAJA/include/RAJA/util/resource.hpp:124:35: error: no type named 'Omp' in namespace 'camp::resources'
    using type = camp::resources::Omp;
                 ~~~~~~~~~~~~~~~~~^
/home/users/sabbott/git/RAJA/include/RAJA/util/resource.hpp:129:35: error: no type named 'Omp' in namespace 'camp::resources'
    using type = camp::resources::Omp;
                 ~~~~~~~~~~~~~~~~~^
/home/users/sabbott/git/RAJA/include/RAJA/util/resource.hpp:134:35: error: no type named 'Omp' in namespace 'camp::resources'
    using type = camp::resources::Omp;
                 ~~~~~~~~~~~~~~~~~^
/home/users/sabbott/git/RAJA/include/RAJA/util/resource.hpp:139:35: error: no type named 'Omp' in namespace 'camp::resources'
    using type = camp::resources::Omp;
                 ~~~~~~~~~~~~~~~~~^
/home/users/sabbott/git/RAJA/include/RAJA/util/resource.hpp:159:47: error: no member named 'Omp' in namespace 'RAJA::resources'
    template <> struct is_resource<resources::Omp> : std::true_type {};

~/git/RAJA/cce_build> git rev-parse --short HEAD
0f96dd476

I can go hop over to camp main and check there as well.

[davidbeckingsale]

@abbotts did you try manually setting the OpenMP offload flags, as done here: https://github.com/LLNL/RAJA/blob/develop/scripts/lc-builds/blueos_clang_omptarget.sh#L42

[abbotts]

@davidbeckingsale , that doesn't seem to help:

~/git/RAJA/cce_build> git rev-parse HEAD
0f96dd4761ee4dcd98ec72660ad0046f9970cf0c
~/git/RAJA/cce_build> git status
On branch develop
Your branch is up to date with 'origin/develop'.

nothing to commit, working tree clean
~/git/RAJA/cce_build> cmake \
-DENABLE_OPENMP=On \
-DENABLE_TARGET_OPENMP=On \
-DENABLE_CUDA=Off \
-DENABLE_CLANG_CUDA=Off \
-DENABLE_HIP=Off  \
-DENABLE_TBB=Off \
-DCMAKE_CXX_COMPILER=CC \
-DCMAKE_C_COMPILER=cc \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_CXX_STANDARD=14 \
-DENABLE_TESTS=On \
-DENABLE_EXAMPLES=Off \
-DOpenMP_CXX_FLAGS="-fopenmp;-fopenmp-targets=amdgcn-amd-amdhsa;-Xopenmp-target=amdgcn-amd-amdhsa -march=gfx90a"  ../

Still gives:

cd /home/users/sabbott/git/RAJA/cce_build/test/integration/plugin && /opt/cray/pe/craype/2.7.17/bin/CC  -DGTEST_HAS_DEATH_TEST=1 -I/home/users/sabbott/git/RAJA/test/include -I/home/users/sabbott/git/RAJA/test/integration/plugin/tests -I/home/users/sabbott/git/RAJA/blt/thirdparty_builtin/googletest-master-2020-01-07/googletest -I/home/users/sabbott/git/RAJA/include -I/home/users/sabbott/git/RAJA/cce_build/include -I/home/users/sabbott/git/RAJA/tpl/camp/include -I/home/users/sabbott/git/RAJA/cce_build/tpl/camp/include -isystem /home/users/sabbott/git/RAJA/blt/thirdparty_builtin/googletest-master-2020-01-07/googletest/include  -Wall -Wextra      -O3 -DNDEBUG -fPIE   -fopenmp -std=c++14 -o CMakeFiles/test-plugin-workgroup-Sequential.exe.dir/test-plugin-workgroup-Sequential.cpp.o -c /home/users/sabbott/git/RAJA/cce_build/test/integration/plugin/test-plugin-workgroup-Sequential.cpp
In file included from /home/users/sabbott/git/RAJA/cce_build/test/integration/plugin/test-plugin-workgroup-Sequential.cpp:11:
In file included from /home/users/sabbott/git/RAJA/test/include/RAJA_test-base.hpp:15:
In file included from /home/users/sabbott/git/RAJA/include/RAJA/RAJA.hpp:44:
In file included from /home/users/sabbott/git/RAJA/include/RAJA/pattern/forall.hpp:64:
In file included from /home/users/sabbott/git/RAJA/include/RAJA/policy/MultiPolicy.hpp:31:
/home/users/sabbott/git/RAJA/include/RAJA/util/resource.hpp:119:35: error: no type named 'Omp' in namespace 'camp::resources'
    using type = camp::resources::Omp;
                 ~~~~~~~~~~~~~~~~~^
/home/users/sabbott/git/RAJA/include/RAJA/util/resource.hpp:124:35: error: no type named 'Omp' in namespace 'camp::resources'
    using type = camp::resources::Omp;
                 ~~~~~~~~~~~~~~~~~^
/home/users/sabbott/git/RAJA/include/RAJA/util/resource.hpp:129:35: error: no type named 'Omp' in namespace 'camp::resources'
    using type = camp::resources::Omp;
                 ~~~~~~~~~~~~~~~~~^
/home/users/sabbott/git/RAJA/include/RAJA/util/resource.hpp:134:35: error: no type named 'Omp' in namespace 'camp::resources'
    using type = camp::resources::Omp;
                 ~~~~~~~~~~~~~~~~~^
/home/users/sabbott/git/RAJA/include/RAJA/util/resource.hpp:139:35: error: no type named 'Omp' in namespace 'camp::resources'
    using type = camp::resources::Omp;
                 ~~~~~~~~~~~~~~~~~^
/home/users/sabbott/git/RAJA/include/RAJA/util/resource.hpp:159:47: error: no member named 'Omp' in namespace 'RAJA::resources'
    template <> struct is_resource<resources::Omp> : std::true_type {};

If that blueos test is working let me double check with AMD's compiler avoiding the Cray CC driver

[abbotts]

Hmmm.. okay, this works:

cmake -DENABLE_OPENMP=On \
-DENABLE_TARGET_OPENMP=On \
-DRAJA_ENABLE_OPENMP=On \
-DRAJA_ENABLE_TARGET_OPENMP=On \
-DENABLE_CUDA=Off \
-DENABLE_CLANG_CUDA=Off \
-DENABLE_HIP=Off  \
-DENABLE_TBB=Off \
-DCMAKE_CXX_COMPILER=CC \
-DCMAKE_C_COMPILER=cc \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_CXX_STANDARD=14 \
-DENABLE_TESTS=On \
-DENABLE_EXAMPLES=Off  ../

If it's intended that CMake options are RAJA_ prefaced, then we can close this.

(And I'll open another bug, because not specifying -DENABLE_OPENMP=On` gives:

CMake Error at cmake/SetupDependentOptions.cmake:19 (message):
  RAJA_ENABLE_OPENMP set to On, but ENABLE_OPENMP is Off.  Please set
  ENABLE_OPENMP to On enable this feature.
Call Stack (most recent call first):
  CMakeLists.txt:110 (include)


-- Configuring incomplete, errors occurred!
See also "/home/users/sabbott/git/RAJA/cce_build/CMakeFiles/CMakeOutput.log".

)

[rhornung67]

@abbotts a few comments:

1. RAJA_ENABLE_OPENMP should not be needed if ENABLE_OPENMP is on. RAJA_ENABLE_OPENMP is a CMake dependent option and so typically would be used to enable OpenMP in a build, but turn it off in RAJA by setting RAJA_ENABLE_OPENMP to off
2. RAJA_ENABLE_TARGET_OPENMP is not a dependent option in RAJA, and is specific to RAJA, since it's not supported in BLT and not in CMake (I think)
3. You shouldn't need to explicitly turn off CUDA, CLANG_CUDA, HIP, and TBB, since they are off by default in RAJA.

@davidbeckingsale these statements are correct, right?

[abbotts]

Okay @rhornung67 , that makes sense and I see it in the documentation, although I still think it would be nice to purge https://github.com/LLNL/RAJA/blob/0f96dd4761ee4dcd98ec72660ad0046f9970cf0c/cmake/SetupRajaConfig.cmake#L59 so ENABLE_TARGET_OPENMP doesn't get used and you get a proper warning:

  Manually-specified variables were not used by the project:

    ENABLE_TARGET_OPENMP

(3) are all off by default. Just a copy-paste from the output of a more generic testing script.

[abbotts]

Closing since this works now when configured with RAJA_ENABLE_TARGET_OPENMP, which is the documented option.