E3SM icon indicating copy to clipboard operation
E3SM copied to clipboard
verified publisher icon E3SM-Project

Cray C++ compiler debug build error (hip backend) - llvm::BitVector::operator: Assertion `Idx < Size && "Out-of-bounds Bit access."' failed.

Open xyuan opened this issue 3 years ago • 6 comments

GPU code build successfully without debug mode, but failed at debug mode on Crusher, see below for details

cd /gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/scream/src/physics/shoc && python3 /gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/Tools/e3sm_compile_wrap.py  /opt/cray/pe/craype/2.7.15/bin/CC -DKOKKOS_DEPENDENCE -DMPICH_SKIP_MPICXX -DSCREAM_CONFIG_IS_CMAKE -DSPDLOG_COMPILED_LIB -D__HIP_ROCclr__ -I/gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/components/eam/src/physics/crm/scream/src/physics/shoc/../share -I/gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/components/eam/src/physics/crm/scream/src -I/gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/scream/src -I/gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/src -I/gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/externals/ekat/src -I/gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/externals/ekat/src/ekat/ekat_f90_modules -I/gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/externals/kokkos -I/gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/externals/kokkos/core/src -I/gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/core/src -I/gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/externals/kokkos/containers/src -I/gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/containers/src -I/gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/externals/kokkos/algorithms/src -I/gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/algorithms/src -I/gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/yaml-cpp/include -I/gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/spdlog/include -Wall   -D__HIP_PLATFORM_HCC__= -D__HIP_PLATFORM_AMD__= -I/opt/rocm-4.5.2/hip/include -I/opt/rocm-4.5.2/llvm/bin/../lib/clang/13.0.0 -I/opt/rocm-4.5.2/include -g -fno-gpu-rdc -x hip --rocm-path=/opt/rocm-4.5.2 --offload-arch=gfx90a -std=gnu++17 -MD -MT scream/src/physics/shoc/CMakeFiles/shoc.dir/shoc_functions_f90.cpp.o -MF CMakeFiles/shoc.dir/shoc_functions_f90.cpp.o.d -o CMakeFiles/shoc.dir/shoc_functions_f90.cpp.o -c /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/components/eam/src/physics/crm/scream/src/physics/shoc/shoc_functions_f90.cpp
No supported cpu target is set, CRAY_CPU_TARGET=x86-64 will be used.
Load a valid targeting module or set CRAY_CPU_TARGET
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/components/eam/src/physics/crm/scream/src/physics/shoc/shoc_functions_f90.cpp:1:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/components/eam/src/physics/crm/scream/src/physics/shoc/shoc_functions_f90.hpp:4:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/components/eam/src/physics/crm/scream/src/share/scream_types.hpp:5:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/src/ekat/kokkos/ekat_kokkos_types.hpp:5:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/src/ekat/kokkos/ekat_kokkos_meta.hpp:4:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/core/src/Kokkos_Core.hpp:51:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/core/src/Kokkos_Core_fwd.hpp:52:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/core/src/Kokkos_Macros.hpp:110:
In file included from /gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/externals/kokkos/KokkosCore_Config_SetupBackend.hpp:47:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/core/src/setup/Kokkos_Setup_HIP.hpp:53:
In file included from /opt/rocm-4.5.2/hip/include/hip/hip_runtime.h:62:
In file included from /opt/rocm-4.5.2/hip/include/hip/amd_detail/amd_hip_runtime.h:85:
/opt/rocm-4.5.2/hip/include/hip/amd_detail/texture_fetch_functions.h:143:5: warning: unused variable 's' [-Wunused-variable]
    TEXTURE_PARAMETERS_INIT;
    ^
/opt/rocm-4.5.2/hip/include/hip/amd_detail/texture_fetch_functions.h:37:42: note: expanded from macro 'TEXTURE_PARAMETERS_INIT'
    unsigned int ADDRESS_SPACE_CONSTANT* s = i + HIP_SAMPLER_OBJECT_OFFSET_DWORD;
                                         ^
/opt/rocm-4.5.2/hip/include/hip/amd_detail/texture_fetch_functions.h:263:5: warning: unused variable 's' [-Wunused-variable]
    TEXTURE_PARAMETERS_INIT;
    ^
/opt/rocm-4.5.2/hip/include/hip/amd_detail/texture_fetch_functions.h:37:42: note: expanded from macro 'TEXTURE_PARAMETERS_INIT'
    unsigned int ADDRESS_SPACE_CONSTANT* s = i + HIP_SAMPLER_OBJECT_OFFSET_DWORD;
                                         ^
/opt/rocm-4.5.2/hip/include/hip/amd_detail/texture_fetch_functions.h:273:5: warning: unused variable 's' [-Wunused-variable]
    TEXTURE_PARAMETERS_INIT;
    ^
/opt/rocm-4.5.2/hip/include/hip/amd_detail/texture_fetch_functions.h:37:42: note: expanded from macro 'TEXTURE_PARAMETERS_INIT'
    unsigned int ADDRESS_SPACE_CONSTANT* s = i + HIP_SAMPLER_OBJECT_OFFSET_DWORD;
                                         ^
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/components/eam/src/physics/crm/scream/src/physics/shoc/shoc_functions_f90.cpp:1:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/components/eam/src/physics/crm/scream/src/physics/shoc/shoc_functions_f90.hpp:4:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/components/eam/src/physics/crm/scream/src/share/scream_types.hpp:5:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/src/ekat/kokkos/ekat_kokkos_types.hpp:5:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/src/ekat/kokkos/ekat_kokkos_meta.hpp:4:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/core/src/Kokkos_Core.hpp:51:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/core/src/Kokkos_Core_fwd.hpp:52:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/core/src/Kokkos_Macros.hpp:110:
In file included from /gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/externals/kokkos/KokkosCore_Config_SetupBackend.hpp:47:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/core/src/setup/Kokkos_Setup_HIP.hpp:53:
In file included from /opt/rocm-4.5.2/hip/include/hip/hip_runtime.h:62:
In file included from /opt/rocm-4.5.2/hip/include/hip/amd_detail/amd_hip_runtime.h:86:
/opt/rocm-4.5.2/hip/include/hip/amd_detail/texture_indirect_functions.h:69:5: warning: unused variable 's' [-Wunused-variable]
    TEXTURE_OBJECT_PARAMETERS_INIT
    ^
/opt/rocm-4.5.2/hip/include/hip/amd_detail/texture_indirect_functions.h:37:42: note: expanded from macro 'TEXTURE_OBJECT_PARAMETERS_INIT'
    unsigned int ADDRESS_SPACE_CONSTANT* s = i + HIP_SAMPLER_OBJECT_OFFSET_DWORD;
                                         ^
/opt/rocm-4.5.2/hip/include/hip/amd_detail/texture_indirect_functions.h:360:5: warning: unused variable 's' [-Wunused-variable]
    TEXTURE_OBJECT_PARAMETERS_INIT
    ^
/opt/rocm-4.5.2/hip/include/hip/amd_detail/texture_indirect_functions.h:37:42: note: expanded from macro 'TEXTURE_OBJECT_PARAMETERS_INIT'
    unsigned int ADDRESS_SPACE_CONSTANT* s = i + HIP_SAMPLER_OBJECT_OFFSET_DWORD;
                                         ^
/opt/rocm-4.5.2/hip/include/hip/amd_detail/texture_indirect_functions.h:488:5: warning: unused variable 's' [-Wunused-variable]
    TEXTURE_OBJECT_PARAMETERS_INIT
    ^
/opt/rocm-4.5.2/hip/include/hip/amd_detail/texture_indirect_functions.h:37:42: note: expanded from macro 'TEXTURE_OBJECT_PARAMETERS_INIT'
    unsigned int ADDRESS_SPACE_CONSTANT* s = i + HIP_SAMPLER_OBJECT_OFFSET_DWORD;
                                         ^
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/components/eam/src/physics/crm/scream/src/physics/shoc/shoc_functions_f90.cpp:1:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/components/eam/src/physics/crm/scream/src/physics/shoc/shoc_functions_f90.hpp:4:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/components/eam/src/physics/crm/scream/src/share/scream_types.hpp:5:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/src/ekat/kokkos/ekat_kokkos_types.hpp:5:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/src/ekat/kokkos/ekat_kokkos_meta.hpp:4:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/core/src/Kokkos_Core.hpp:51:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/core/src/Kokkos_Core_fwd.hpp:52:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/core/src/Kokkos_Macros.hpp:110:
In file included from /gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/externals/kokkos/KokkosCore_Config_SetupBackend.hpp:47:
In file included from /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/core/src/setup/Kokkos_Setup_HIP.hpp:53:
In file included from /opt/rocm-4.5.2/hip/include/hip/hip_runtime.h:62:
/opt/rocm-4.5.2/hip/include/hip/amd_detail/amd_hip_runtime.h:296:16: warning: attribute declaration must precede definition [-Wignored-attributes]
__attribute__((weak))
               ^
/opt/rocm-4.5.2/hip/include/hip/amd_detail/amd_hip_runtime.h:286:24: note: previous definition is here
  static constexpr __X x{};
                       ^
/opt/rocm-4.5.2/hip/include/hip/amd_detail/amd_hip_runtime.h:301:16: warning: attribute declaration must precede definition [-Wignored-attributes]
__attribute__((weak))
               ^
/opt/rocm-4.5.2/hip/include/hip/amd_detail/amd_hip_runtime.h:287:24: note: previous definition is here
  static constexpr __Y y{};
                       ^
/opt/rocm-4.5.2/hip/include/hip/amd_detail/amd_hip_runtime.h:306:16: warning: attribute declaration must precede definition [-Wignored-attributes]
__attribute__((weak))
               ^
/opt/rocm-4.5.2/hip/include/hip/amd_detail/amd_hip_runtime.h:288:24: note: previous definition is here
  static constexpr __Z z{};
                       ^
clang-14: /workspace/llvm/include/llvm/ADT/BitVector.h:438: llvm::BitVector::reference llvm::BitVector::operator[](unsigned int): Assertion `Idx < Size && "Out-of-bounds Bit access."' failed.
PLEASE submit a bug report to Cray and include the crash backtrace, preprocessed source, and associated run script.

Stack dump:

0.	Program arguments: /opt/cray/pe/cce/14.0.0/cce-clang/x86_64/bin/clang-14 -cc1 -triple amdgcn-amd-amdhsa -mllvm -cray-omp-opt-fork-call -mllvm -cray-omp-parallel-opt-call -mllvm -cray-openmp-rename-outlined -fcray-gpu -mllvm -cray-enhanced-asm=1 -fenhanced-asm=1 -mllvm -cray-enhanced-ir=1 -fenhanced-ir=1 -fomp-local-offload-table -ffortran -aux-triple x86_64-unknown-linux-gnu -emit-obj -mrelax-all --mrelax-relocations -disable-free -clear-ast-before-backend -main-file-name shoc_functions_f90.cpp -mrelocation-model pic -pic-level 1 -fhalf-no-semantic-interposition -mframe-pointer=all -fno-rounding-math -mconstructor-aliases -aux-target-cpu x86-64 -fcuda-is-device -mllvm -amdgpu-internalize-symbols -fcuda-allow-variadic-functions -fvisibility hidden -fapply-global-visibility-to-externs -mlink-builtin-bitcode /opt/rocm-4.5.2/amdgcn/bitcode/hip.bc -mlink-builtin-bitcode /opt/rocm-4.5.2/amdgcn/bitcode/ocml.bc -mlink-builtin-bitcode /opt/rocm-4.5.2/amdgcn/bitcode/ockl.bc -mlink-builtin-bitcode /opt/rocm-4.5.2/amdgcn/bitcode/oclc_daz_opt_off.bc -mlink-builtin-bitcode /opt/rocm-4.5.2/amdgcn/bitcode/oclc_unsafe_math_off.bc -mlink-builtin-bitcode /opt/rocm-4.5.2/amdgcn/bitcode/oclc_finite_only_off.bc -mlink-builtin-bitcode /opt/rocm-4.5.2/amdgcn/bitcode/oclc_correctly_rounded_sqrt_on.bc -mlink-builtin-bitcode /opt/rocm-4.5.2/amdgcn/bitcode/oclc_wavefrontsize64_on.bc -mlink-builtin-bitcode /opt/rocm-4.5.2/amdgcn/bitcode/oclc_isa_version_90a.bc -target-cpu gfx90a -mllvm -treat-scalable-fixed-error-as-warning -debug-info-kind=constructor -dwarf-version=5 -debugger-tuning=gdb -resource-dir /opt/cray/pe/cce/14.0.0/cce-clang/x86_64/lib/clang/14.0.0 -dependency-file CMakeFiles/shoc.dir/shoc_functions_f90.cpp.o.d -MT scream/src/physics/shoc/CMakeFiles/shoc.dir/shoc_functions_f90.cpp.o -sys-header-deps -internal-isystem /opt/cray/pe/cce/14.0.0/cce-clang/x86_64/lib/clang/14.0.0/include/cuda_wrappers -internal-isystem /opt/rocm-4.5.2/include -include __clang_hip_runtime_wrapper.h -isystem /opt/cray/pe/cce/14.0.0/cce-clang/x86_64/lib/clang/14.0.0/include -isystem /opt/cray/pe/cce/14.0.0/cce/x86_64/include/craylibs -D __CRAY_X86_64 -D __CRAY_AMD_GFX90A -D __CRAYXT_COMPUTE_LINUX_TARGET -D KOKKOS_DEPENDENCE -D MPICH_SKIP_MPICXX -D SCREAM_CONFIG_IS_CMAKE -D SPDLOG_COMPILED_LIB -D __HIP_ROCclr__ -I /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/components/eam/src/physics/crm/scream/src/physics/shoc/../share -I /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/components/eam/src/physics/crm/scream/src -I /gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/scream/src -I /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/src -I /gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/externals/ekat/src -I /gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/externals/ekat/src/ekat/ekat_f90_modules -I /gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/externals/kokkos -I /gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/externals/kokkos/core/src -I /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/core/src -I /gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/externals/kokkos/containers/src -I /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/containers/src -I /gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/externals/kokkos/algorithms/src -I /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/kokkos/algorithms/src -I /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/yaml-cpp/include -I /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/externals/ekat/extern/spdlog/include -D __HIP_PLATFORM_HCC__= -D __HIP_PLATFORM_AMD__= -I /opt/rocm-4.5.2/hip/include -I /opt/rocm-4.5.2/llvm/bin/../lib/clang/13.0.0 -I /opt/rocm-4.5.2/include -I /opt/cray/pe/libsci/21.08.1.2/CRAY/9.0/x86_64/include -I /opt/cray/pe/mpich/8.1.16/ofi/cray/10.0/include -I /opt/cray/pe/parallel-netcdf/1.12.1.7/crayclang/10.0/include -I /opt/cray/pe/netcdf-hdf5parallel/4.7.4.7/crayclang/10.0/include -I /opt/cray/pe/hdf5-parallel/1.12.0.7/crayclang/10.0/include -I /opt/cray/pe/dsmml/0.2.2/dsmml//include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++ -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/x86_64-suse-linux -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/backward -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++ -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/x86_64-suse-linux -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/backward -internal-isystem /opt/cray/pe/cce/14.0.0/cce-clang/x86_64/lib/clang/14.0.0/include -internal-isystem /usr/local/include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../x86_64-suse-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /opt/cray/pe/cce/14.0.0/cce-clang/x86_64/lib/clang/14.0.0/include -internal-isystem /usr/local/include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../x86_64-suse-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -Wall -std=gnu++17 -fdeprecated-macro -fno-autolink -fdebug-compilation-dir=/gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld/scream/src/physics/shoc -ferror-limit 19 -fhip-new-launch-api -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -cuid=9ecccd4a88b8bf07 -fcuda-allow-variadic-functions -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/shoc_functions_f90-b7665e/shoc_functions_f90-gfx90a.o -x hip /gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/components/eam/src/physics/crm/scream/src/physics/shoc/shoc_functions_f90.cpp
1.	<eof> parser at end of file
2.	Code generation
3.	Running pass 'CallGraph Pass Manager' on module '/gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher/components/eam/src/physics/crm/scream/src/physics/shoc/shoc_functions_f90.cpp'.
4.	Running pass 'SI lower SGPR spill instructions' on function '@_ZN6scream4shoc9FunctionsIdN6Kokkos6DeviceINS2_12Experimental3HIPENS4_8HIPSpaceEEEE24diag_second_shoc_momentsERKNS2_4Impl13HIPTeamMemberERKiSE_RKNS2_4ViewIPKN4ekat4PackIdLi1EEEJNS2_11LayoutRightES7_NS2_12MemoryTraitsILj1EEEEEESQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_SQ_RKdSS_SS_SS_RdST_NSG_16WorkspaceManagerISI_S7_E9WorkspaceERKNSF_IPSI_JSL_S7_SN_EEES10_S10_S10_S10_S10_S10_S10_S10_'
 #0 0x00000000038208bf PrintStackTraceSignalHandler(void*) Signals.cpp:0:0
 #1 0x000000000381d9dd SignalHandler(int) Signals.cpp:0:0
 #2 0x00007fa076c758bf (/lib64/libpthread.so.0+0x168bf)
 #3 0x00007fa075b70cda raise (/lib64/libc.so.6+0x4acda)
 #4 0x00007fa075b72374 abort (/lib64/libc.so.6+0x4c374)
 #5 0x00007fa075b68cd9 __assert_fail_base (/lib64/libc.so.6+0x42cd9)
 #6 0x00007fa075b68d61 __assert_fail (/lib64/libc.so.6+0x42d61)
 #7 0x000000000123902e (anonymous namespace)::SILowerSGPRSpills::runOnMachineFunction(llvm::MachineFunction&) SILowerSGPRSpills.cpp:0:0
 #8 0x00000000028b69cf llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/opt/cray/pe/cce/14.0.0/cce-clang/x86_64/bin/clang-14+0x28b69cf)
 #9 0x0000000002de3540 llvm::FPPassManager::runOnFunction(llvm::Function&) (/opt/cray/pe/cce/14.0.0/cce-clang/x86_64/bin/clang-14+0x2de3540)
#10 0x00000000024c02d9 (anonymous namespace)::CGPassManager::runOnModule(llvm::Module&) CallGraphSCCPass.cpp:0:0
#11 0x0000000002de5196 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/opt/cray/pe/cce/14.0.0/cce-clang/x86_64/bin/clang-14+0x2de5196)
#12 0x0000000003ba670a clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, clang::FrontendOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream> >) (/opt/cray/pe/cce/14.0.0/cce-clang/x86_64/bin/clang-14+0x3ba670a)
#13 0x0000000004a7b0b5 clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/opt/cray/pe/cce/14.0.0/cce-clang/x86_64/bin/clang-14+0x4a7b0b5)
#14 0x00000000055f7168 clang::ParseAST(clang::Sema&, bool, bool) (/opt/cray/pe/cce/14.0.0/cce-clang/x86_64/bin/clang-14+0x55f7168)
#15 0x0000000004362cf8 clang::FrontendAction::Execute() (/opt/cray/pe/cce/14.0.0/cce-clang/x86_64/bin/clang-14+0x4362cf8)
#16 0x00000000042e2b65 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/opt/cray/pe/cce/14.0.0/cce-clang/x86_64/bin/clang-14+0x42e2b65)
#17 0x0000000004429fca clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/opt/cray/pe/cce/14.0.0/cce-clang/x86_64/bin/clang-14+0x4429fca)
#18 0x0000000000df10a5 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/opt/cray/pe/cce/14.0.0/cce-clang/x86_64/bin/clang-14+0xdf10a5)
#19 0x0000000000dee7ab ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&) driver.cpp:0:0
#20 0x0000000000d2d936 main (/opt/cray/pe/cce/14.0.0/cce-clang/x86_64/bin/clang-14+0xd2d936)
#21 0x00007fa075b5b2bc __libc_start_main (/lib64/libc.so.6+0x352bc)
#22 0x0000000000debe39 _start /home/abuild/rpmbuild/BUILD/glibc-2.26/csu/../sysdeps/x86_64/start.S:120:0
clang-14: error: unable to execute command: Aborted
clang-14: error: clang frontend command failed due to signal (use -v to see invocation)
Cray clang version 14.0.0  (c98838affc7b58fed2a72f164d77c35e1bc8772f)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/cray/pe/cce/14.0.0/cce-clang/x86_64/share/../bin
clang-14: note: diagnostic msg: Error generating preprocessed source(s).
Target CMakeFiles/shoc.dir/shoc_functions_f90.cpp.o built in 12.370942 seconds
make[2]: *** [scream/src/physics/shoc/CMakeFiles/shoc.dir/build.make:93: scream/src/physics/shoc/CMakeFiles/shoc.dir/shoc_functions_f90.cpp.o] Error 254
make[2]: Leaving directory '/gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld'
make[1]: *** [CMakeFiles/Makefile2:1812: scream/src/physics/shoc/CMakeFiles/shoc.dir/all] Error 2
make[1]: Leaving directory '/gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.crayclanggpu.4x1/bld/cmake-bld'
make: *** [Makefile:159: all] Error 2

xyuan avatar Jun 05 '22 15:06 xyuan

cc @abbotts

sarats avatar Jun 06 '22 17:06 sarats

Can we reproduce this issue with standalone SHOC in debug mode build and post the instructions?

sarats avatar Jun 14 '22 18:06 sarats

Because this is a C++ issue while creating an object file, we should be able to just capture the preprocessed source and use it to open an HPE issue. The compiler tried to do this automatically but failed:

clang-14: note: diagnostic msg: Error generating preprocessed source(s).

That happens sometimes.

The first thing to try is to copy/paste the specific compilation command, add --save-temps -v, and capture the output. Then look for a file called shoc_functions_f90-hip-amdgcn-amd-amdhsa-gf90a.cui, which should be the fully preprocessed source that was passed into the HIP compilation pass.

With that fully preprocessed .cui file and the verbose compilation output, I should be able to reproduce the compiler error. The cui file will be quite large and we may need to transfer it on the OLCF systems.

Sometimes adding --save-temps -v can change the compilation trajectory and either catch a different bug or avoid the problem. If either of those happens there's some other things we can try.

abbotts avatar Jun 14 '22 20:06 abbotts

@xyuan Can you try Steve's suggestion above to create a reproducer?

sarats avatar Jun 29 '22 22:06 sarats

I generated the cui files, and it is located here,

[[email protected] shoc]$ pwd /gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.amdclanggpu.16x1.08/bld/cmake-bld/scream/src/physics/shoc [[email protected] shoc]$ ls -al total 5108 drwxr-xr-x 3 yuanx yuanx 4096 Jul 11 14:49 . drwxr-xr-x 6 yuanx yuanx 4096 Jun 14 09:52 .. drwxr-xr-x 3 yuanx yuanx 4096 Jun 14 09:52 CMakeFiles -rw-r--r-- 1 yuanx yuanx 1325 Jun 14 09:52 cmake_install.cmake -rw-r--r-- 1 yuanx yuanx 458 Jun 14 09:52 CTestTestfile.cmake -rw-r--r-- 1 yuanx yuanx 19262 Jun 14 09:52 Makefile -rw-r--r-- 1 yuanx yuanx 1192968 Jul 11 14:48 shoc_functions_f90-hip-amdgcn-amd-amdhsa-gfx90a.bc -rw-r--r-- 1 yuanx yuanx 5190756 Jul 11 14:48 shoc_functions_f90-hip-amdgcn-amd-amdhsa-gfx90a.cui

xyuan avatar Jul 11 '22 18:07 xyuan

Just getting around to this now. @xyuan I don't have permissions to access those files. Could you move them to the project's world-shared? Then I'll pull them off and get a ticket open.

abbotts avatar Jul 26 '22 16:07 abbotts