Check for working CUDA compiler fails (gcc8.2 - cuda10.1.243)

[rrsettgast]

When trying to run cmake for various BLT projects (GEOSX, RAJA, CHAI) I get the following error:

-- Check for working CUDA compiler: /data_local/sw/cuda/10.1.243/bin/nvcc
-- Check for working CUDA compiler: /data_local/sw/cuda/10.1.243/bin/nvcc -- broken
CMake Error at /data_local/sw/spack/0.12.1/opt/spack/linux-rhel7-ppc64le/gcc-4/cmake-3.12-7v2vmul54euxqwmdlmqzdfbb2zojvkcl/share/cmake-3.12/Modules/CMakeTestCUDACompiler.cmake:46 (message):
  The CUDA compiler

    "/data_local/sw/cuda/10.1.243/bin/nvcc"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /work206/workrd/SCR/GEOS/GEOSX/thirdPartyLibs/build-pangea3-debug/CMakeFiles/CMakeTmp
    Run Build Command:"/usr/bin/gmake" "cmTC_0fd95/fast"
    /usr/bin/gmake -f CMakeFiles/cmTC_0fd95.dir/build.make CMakeFiles/cmTC_0fd95.dir/build
    gmake[1]: Entering directory `/work206/workrd/SCR/GEOS/GEOSX/thirdPartyLibs/build-pangea3-debug/CMakeFiles/CMakeTmp'
    Building CUDA object CMakeFiles/cmTC_0fd95.dir/main.cu.o
    /data_local/sw/cuda/10.1.243/bin/nvcc -ccbin=/data_local/sw/spectrum_mpi/10.03.00.01rtm3-rh7_20190611/bin/mpicxx   -restrict -arch sm_70 --expt-extended-lambda -Werror cross-execution-space-call,reorder,deprecated-declarations    -x cu -c /work206/workrd/SCR/GEOS/GEOSX/thirdPartyLibs/build-pangea3-debug/CMakeFiles/CMakeTmp/main.cu -o CMakeFiles/cmTC_0fd95.dir/main.cu.o
    /data_local/sw/spack/opt/spack/linux-rhel7-ppc64le/gcc-4.8.5/gcc-8.2.0-suea3az6vbveib7t7d7roevy5y2qeebb/include/c++/8.2.0/type_traits(335): error: identifier "__ieee128" is undefined
    
    /data_local/sw/spack/opt/spack/linux-rhel7-ppc64le/gcc-4.8.5/gcc-8.2.0-suea3az6vbveib7t7d7roevy5y2qeebb/include/c++/8.2.0/bits/std_abs.h(101): error: identifier "__ieee128" is undefined
    
    /data_local/sw/spack/opt/spack/linux-rhel7-ppc64le/gcc-4.8.5/gcc-8.2.0-suea3az6vbveib7t7d7roevy5y2qeebb/include/c++/8.2.0/bits/std_abs.h(102): error: identifier "__ieee128" is undefined
    
    3 errors detected in the compilation of "/tmp/tmpxft_0001e38f_00000000-6_main.cpp1.ii".
    gmake[1]: *** [CMakeFiles/cmTC_0fd95.dir/main.cu.o] Error 1
    gmake[1]: Leaving directory `/work206/workrd/SCR/GEOS/GEOSX/thirdPartyLibs/build-pangea3-debug/CMakeFiles/CMakeTmp'
    gmake: *** [cmTC_0fd95/fast] Error 2

The "workaround" was to specify -std=c++11 in CMAKE_CUDA_FLAGS within my hostconfig, which suppresses the inclusion of template specializations involving __float128, which is apparently __ieee128. Oh...and the same for RAJA and CHAI. Of course, I had already specified CMAKE_CXX_STANDARD 14, but that didn't make it to the command line of the compiler check for some reason? Perhaps this should be considered at gcc bug?

[davidbeckingsale]

@rrsettgast - this is a problem with GCC and CUDA 10, fixed in CUDA 11. The correct workaround is to add the flag -Xcompiler -mno-float128 to your CUDA compile flags. We already have the workaround for this in RAJA: https://github.com/LLNL/RAJA/blob/develop/CMakeLists.txt#L92

[rrsettgast]

Thanks @davidbeckingsale! Of course we are using RAJA 0.10.1. Is there an equivalent workaround in CHAI? I couldn't find it.

[rrsettgast]

@davidbeckingsale Still get the error

-- Check for working CUDA compiler: /data_local/sw/cuda/10.1.243/bin/nvcc -- broken
CMake Error at /data_local/sw/spack/0.12.1/opt/spack/linux-rhel7-ppc64le/gcc-4/cmake-3.12-7v2vmul54euxqwmdlmqzdfbb2zojvkcl/share/cmake-3.12/Modules/CMakeTestCUDACompiler.cmake:46 (message):
  The CUDA compiler

    "/data_local/sw/cuda/10.1.243/bin/nvcc"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /work206/workrd/SCR/GEOS/GEOSX/GEOSX_Clone01/build-pangea3-release/CMakeFiles/CMakeTmp
    
    Run Build Command:"/usr/bin/gmake" "cmTC_1d2b8/fast"
    /usr/bin/gmake -f CMakeFiles/cmTC_1d2b8.dir/build.make CMakeFiles/cmTC_1d2b8.dir/build
    gmake[1]: Entering directory `/work206/workrd/SCR/GEOS/GEOSX/GEOSX_Clone01/build-pangea3-release/CMakeFiles/CMakeTmp'
    Building CUDA object CMakeFiles/cmTC_1d2b8.dir/main.cu.o
    /data_local/sw/cuda/10.1.243/bin/nvcc -ccbin=/data_local/sw/spectrum_mpi/10.03.00.01rtm3-rh7_20190611/bin/mpicxx -DcmTC_1d2b8_EXPORTS  -restrict -arch sm_70 --expt-extended-lambda -Werror cross-execution-space-call,reorder,deprecated-declarations -Xcompiler -mno-float128    -x cu -c /work206/workrd/SCR/GEOS/GEOSX/GEOSX_Clone01/build-pangea3-release/CMakeFiles/CMakeTmp/main.cu -o CMakeFiles/cmTC_1d2b8.dir/main.cu.o
    /data_local/sw/spack/opt/spack/linux-rhel7-ppc64le/gcc-4.8.5/gcc-8.2.0-suea3az6vbveib7t7d7roevy5y2qeebb/include/c++/8.2.0/type_traits(335): error: identifier "__float128" is undefined
    
    /data_local/sw/spack/opt/spack/linux-rhel7-ppc64le/gcc-4.8.5/gcc-8.2.0-suea3az6vbveib7t7d7roevy5y2qeebb/include/c++/8.2.0/bits/std_abs.h(101): error: identifier "__float128" is undefined
    
    /data_local/sw/spack/opt/spack/linux-rhel7-ppc64le/gcc-4.8.5/gcc-8.2.0-suea3az6vbveib7t7d7roevy5y2qeebb/include/c++/8.2.0/bits/std_abs.h(102): error: identifier "__float128" is undefined
    
    3 errors detected in the compilation of "/tmp/tmpxft_00014bd6_00000000-6_main.cpp1.ii".
    gmake[1]: *** [CMakeFiles/cmTC_1d2b8.dir/main.cu.o] Error 1
    gmake[1]: Leaving directory `/work206/workrd/SCR/GEOS/GEOSX/GEOSX_Clone01/build-pangea3-release/CMakeFiles/CMakeTmp'
    gmake: *** [cmTC_1d2b8/fast] Error 2

Looking at the offending files:

bits/std_abs.h:#if !defined(__STRICT_ANSI__) && defined(_GLIBCXX_USE_FLOAT128)
bits/std_abs.h-  inline _GLIBCXX_CONSTEXPR
bits/std_abs.h-  __float128
bits/std_abs.h-  abs(__float128 __x)
bits/std_abs.h-  { return __x < 0 ? -__x : __x; }
bits/std_abs.h-#endif

type_traits:#if !defined(__STRICT_ANSI__) && defined(_GLIBCXX_USE_FLOAT128)
type_traits-  template<>
type_traits-    struct __is_floating_point_helper<__float128>
type_traits-    : public true_type { };
type_traits-#endif

and _GLIBCXX_USE_FLOAT128 is defined in: /data_local/sw/spack/opt/spack/linux-rhel7-ppc64le/gcc-4.8.5/gcc-8.2.0-suea3az6vbveib7t7d7roevy5y2qeebb/include/c++/8.2.0/powerpc64le-unknown-linux-gnu/bits/c++config.h

/* Define if __float128 is supported on this host. */
#define _GLIBCXX_USE_FLOAT128 1

So I think that -mno-float128 may be a gcc configuration flag rather than a compilation option??

At any rate, I am thinking this is not a BLT issue at all. Correct?

[klevzoff]

@rrsettgast - this is a problem with GCC and CUDA 10, fixed in CUDA 11. The correct workaround is to add the flag -Xcompiler -mno-float128 to your CUDA compile flags. We already have the workaround for this in RAJA: https://github.com/LLNL/RAJA/blob/develop/CMakeLists.txt#L92

Hi @davidbeckingsale ,

Just wanted to point out something I came across. -mno-float128 is a PowerPC-specific gcc option and is not recognized on x86_64 (at least as of gcc-8.4.0), leading to compilation errors. Perhaps it should additionally be guarded by something like

if (CMAKE_SYSTEM_PROCESSOR MATCHES "^(powerpc|ppc)64.*$")