RAJA
RAJA copied to clipboard
LLNL
Fix "exhaustive" atomic tests
The tests: test-forall-atomic-basic-OpenMP.exe test-forall-AtomicRefMinMax-OpenMP.exe
fail when we build and run exhaustive tests. They work with the default test builds.
@rchen20 assigning this to you to look at when you have time. You won't see these test failures on the develop branch so they are likely due to me adding in new OpenMP execution policies to test. A PR for these changes is coming soon.
@rchen20 is this still an issue? I tried a couple of compilers (gnu, clang) and everything passed.
I'll try this with XL because that's usually the most problematic compiler.
Strange, I'm running in to 2 different compilation issues with test-forall-atomic-basic-OpenMP.exe. One is the excessive recursion on our templating (saw this in https://github.com/LLNL/RAJA/pull/1165), the other is the camp::resources::Omp not found issue. I'll need to dig further in to why these are happening.
nvcc/10.1.243 + xl/2021.12.22 -
cd /usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-nvcc10.1.243-sm_70-xl2021.12.22/test/functional/forall/atomic-basic && /usr/tce/packages/cuda/cuda-10.1.243/bin/nvcc -ccbin=/usr/tce/packages/xl/xl-2021.12.22/bin/xlc++_r -DGTEST_HAS_DEATH_TEST=1 -I/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/test/include -I/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/test/functional/forall/atomic-basic/tests -I/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/blt/thirdparty_builtin/googletest-master-2020-01-07/googletest -I/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/include -I/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-nvcc10.1.243-sm_70-xl2021.12.22/include -I/usr/tce/packages/cuda/cuda-10.1.243/include -I/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/tpl/camp/include -I/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-nvcc10.1.243-sm_70-xl2021.12.22/tpl/camp/include -isystem=/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/blt/thirdparty_builtin/googletest-master-2020-01-07/googletest/include -isystem=/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/tpl/cub -restrict -arch sm_70 --expt-extended-lambda --expt-relaxed-constexpr -Xcudafe "--display_error_number" -O3 -Xcompiler -O3 -Xcompiler -qxlcompatmacros -Xcompiler -qalias=noansi -Xcompiler -qsmp=omp -Xcompiler -qhot -Xcompiler -qnoeh -Xcompiler -qsuppress=1500-029 -Xcompiler -qsuppress=1500-036 -Xcompiler=-fPIE -Xcompiler=-qsmp=omp -std=c++14 -x cu -c /usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-nvcc10.1.243-sm_70-xl2021.12.22/test/functional/forall/atomic-basic/test-forall-atomic-basic-OpenMP.cpp -o CMakeFiles/test-forall-atomic-basic-OpenMP.exe.dir/test-forall-atomic-basic-OpenMP.cpp.o
/usr/workspace/wsrzc/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/blt/thirdparty_builtin/googletest-master-2020-01-07/googletest/include/gtest/internal/gtest-internal.h(731): error #456: excessive recursion at instantiation of function "testing::internal::TypeParameterizedTest<Fixture, TestSel, Types>::Register [with Fixture=ForallAtomicBasicTest, TestSel=testing::internal::TemplateSel<gtest_suite_ForallAtomicBasicTest_::AtomicBasicForall>, Types=testing::internal::Types<camp::list<RAJA::policy::omp::omp_parallel_for_guided_exec<-1>, RAJA::policy::cuda::cuda_atomic_explicitRAJA::builtin_atomic, camp::resources::v1::Host, RAJA::Index_type, unsigned long long>, camp::list<RAJA::policy::omp::omp_parallel_for_guided_exec<-1>, RAJA::policy::cuda::cuda_atomic_explicitRAJA::builtin_atomic, camp::resources::v1::Host, RAJA::Index_type, float>, camp::list<RAJA::policy::omp::omp_parallel_for_guided_exec<-1>
xl/2021.12.22_omptarget -
[ 66%] Building CXX object test/functional/forall/atomic-basic/CMakeFiles/test-forall-atomic-basic-OpenMP.exe.dir/test-forall-atomic-basic-OpenMP.cpp.o cd /usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-xl_omptarget-2021.12.22/test/functional/forall/atomic-basic && /usr/tce/packages/xl/xl-2021.12.22/bin/xlc++_r -+ -DGTEST_HAS_DEATH_TEST=1 -I/usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/test/include -I/usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/test/functional/forall/atomic-basic/tests -I/usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/blt/thirdparty_builtin/googletest-master-2020-01-07/googletest -I/usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/include -I/usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-xl_omptarget-2021.12.22/include -I/usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/tpl/camp/include -I/usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-xl_omptarget-2021.12.22/tpl/camp/include -I/usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/blt/thirdparty_builtin/googletest-master-2020-01-07/googletest/include -qthreaded -std=c++14 -O3 -qxlcompatmacros -qlanglvl=extended0x -qalias=noansi -qsmp=omp -qhot -qpic -qsuppress=1500-029 -qsuppress=1500-036 -qpic -qoffload -qsmp=omp -qalias=noansi -std=c++1y -o CMakeFiles/test-forall-atomic-basic-OpenMP.exe.dir/test-forall-atomic-basic-OpenMP.cpp.o -c /usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-xl_omptarget-2021.12.22/test/functional/forall/atomic-basic/test-forall-atomic-basic-OpenMP.cpp In file included from /usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/build_lc_blueos-xl_omptarget-2021.12.22/test/functional/forall/atomic-basic/test-forall-atomic-basic-OpenMP.cpp:11: In file included from /usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/test/include/RAJA_test-base.hpp:15: In file included from /usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/include/RAJA/RAJA.hpp:44: In file included from /usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/include/RAJA/pattern/forall.hpp:64: In file included from /usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/include/RAJA/policy/MultiPolicy.hpp:31: /usr/WS1/chen59/allraja/rajaatomicexhaustive/raja_git_atomicexhaustive/include/RAJA/util/resource.hpp:119:35: error: no type named 'Omp' in namespace 'camp::resources' using type = camp::resources::Omp; ~~~~~~~~~~~~~~~~~^
FWIW, I tried compiling earlier with multiple different versions of clang-ibm and OpenMP target enabled and saw a bunch of similar errors about no 'Omp' in 'camp::resources' namespace. I recall that there was an issue where ENABLE_OPENMP was not getting propagated to camp and I thought it was fixed a while back.
The compilation that I reported as successful above was on toss3. I just built with clang-ibm-12.0.1 on rzansel and everything compiled, but the OpenMP sort tests failed with segfaults. All tests pass with a couple of vanilla clang compiler versions I tried on rzansel. Build errors in our examples with xl/2021.12.22 coming from camp's tuple.hpp file -- unable to access private class members.
Is there something messed up in the CMake logic that prevents camp from having OpenMP enabled in some cases, are we not passing the right compiler flags for XL, or are XL compilers broken?
Adding -DCAMP_ENABLE_TARGET_OPENMP looks like it's helping; I'm just waiting for XL to build it . . .
Once that's confirmed to work, I'll put a PR up. Still not sure what we can do about the excessive recursion in the compiler though.
should we add something to our CMake to set that flag when OpenMP target is enabled in RAJA?
You can probably copy and tweak this snippet from Umpire:
https://github.com/LLNL/Umpire/blob/develop/src/umpire/tpl/CMakeLists.txt#L92-L97
Thanks @davidbeckingsale, that worked for the clang-omptarget tests. Still waiting on XL to see if the test works, but made a PR for this anyway because we should have it https://github.com/LLNL/RAJA/pull/1207.