pumi-pic
pumi-pic copied to clipboard
SCOREC
build error on with Kokkos >= 4.7.00
On OLCF Frontier using Kokkos 4.7.99 (develop @ 0babcae5a) I'm hitting the following compilation error:
pumi-pic/support/SupportKK.h:163:9: error: no matching function for call to object of type 'const Kokkos::View<double (*)[3], Kokkos::LayoutLeft, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>>'
163 | new_view(i) = view(start + i);
| ^~~~~~~~
and on SCOREC RHEL9 with an NVIDIA 3060:
/users/smithc11/develop/pumipicDev/pumi-pic/support/SupportKK.h(163): error: no instance of function template "Kokkos::View<DataType, Properties...>::operator() [with DataType=Vector3 *, Properties=<Kokkos::LayoutLeft, Kokkos::CudaSpace::device_type>]" matches the argument list
argument types are: (const int)
object type is: const Kokkos::View<Vector3 *, Kokkos::LayoutLeft, Kokkos::CudaSpace::device_type>
new_view(i) = view(start + i);
^
SCOREC RHEL9 - nvidia 3060
env
module use /opt/scorec/spack/rhel9/v0201_4/lmod/linux-rhel9-x86_64/Core/
module load \
gcc/12.3.0-iil3lno \
cmake/3.26.3-2duxfcd \
mpich/4.1.1-xpoyz4t \
cuda/12.1.1-zxa4msk
kokkos cmake configure
cmake -S $root/kokkos -B buildKokkos \
-DCMAKE_CXX_COMPILER=$root/kokkos/bin/nvcc_wrapper \
-DKokkos_ARCH_AMPERE86=ON \
-DKokkos_ENABLE_SERIAL=ON \
-DKokkos_ENABLE_OPENMP=off \
-DKokkos_ENABLE_CUDA=on \
-DKokkos_ENABLE_CUDA_LAMBDA=on \
-DKokkos_ENABLE_DEBUG=on \
-DKokkos_ENABLE_PROFILING=on \
-DCMAKE_INSTALL_PREFIX=buildKokkos/install
pumipic cmake configure
cmake -S $root/pumi-pic -B buildPumpic \
-DCMAKE_INSTALL_PREFIX=buildPumipic/install \
-DCMAKE_CXX_COMPILER=mpicxx \
-DIS_TESTING=ON \
-DPS_IS_TESTING=ON \
-DTEST_DATA_DIR=$root/pumi-pic/pumipic-data \
-DOmega_h_PREFIX=$oh \
-DKokkos_PREFIX=$kk \
-DEnGPar_PREFIX=$engpar
verbose make output
[ 10%] Building CXX object particle_structs/CMakeFiles/smoke_test_particle.dir/test/smoke_test_particle.cpp.o
cd /users/smithc11/develop/pumipicDev/builds/AMPERE86/pumipic/particle_structs && /users/smithc11/develop/pumipicDev/builds/AMPERE86/kokkos/install/bin/kokkos_launch_compiler /users/smithc11/develop/pumipicDev/builds/AMPERE86/kokkos/install/bin/nvcc_wrapper /opt/scorec/spack/rhel9/v0201_4/install/linux-rhel9-x86_64/gcc-12.3.0/mpich-4.1.1-xpoyz4tqgfxtrm6m7qq67q4ccp5pnlre/bin/mpicxx /opt/scorec/spack/rhel9/v0201_4/install/linux-rhel9-x86_64/gcc-12.3.0/mpich-4.1.1-xpoyz4tqgfxtrm6m7qq67q4ccp5pnlre/bin/mpicxx -DKOKKOS_DEPENDENCE -DKOKKOS_ENABLED -DPP_USE_CUDA -DPP_USE_GPU -DPUMIPIC_HAS_MALLINFO2 -DPUMIPIC_PRINT_ENABLED -I/users/smithc11/develop/pumipicDev/pumi-pic/support -I/users/smithc11/develop/pumipicDev/pumi-pic/particle_structs/src -I/users/smithc11/develop/pumipicDev/pumi-pic/particle_structs/src/support -I/users/smithc11/develop/pumipicDev/pumi-pic/particle_structs/src/scs -I/users/smithc11/develop/pumipicDev/pumi-pic/particle_structs/src/csr -I/users/smithc11/develop/pumipicDev/pumi-pic/particle_structs/src/cabm -I/users/smithc11/develop/pumipicDev/pumi-pic/particle_structs/src/dps -I/users/smithc11/develop/pumipicDev/pumi-pic/support/support -isystem /users/smithc11/develop/pumipicDev/builds/AMPERE86/kokkos/install/include -isystem /opt/scorec/spack/rhel9/v0201_4/install/linux-rhel9-x86_64/gcc-12.3.0/cuda-12.1.1-zxa4mskqvbkiefzkvnuatlq7skxjnzt6/include -std=c++17 -extended-lambda -Wext-lambda-captures-this -lineinfo -arch=sm_86 -MD -MT particle_structs/CMakeFiles/smoke_test_particle.dir/test/smoke_test_particle.cpp.o -MF CMakeFiles/smoke_test_particle.dir/test/smoke_test_particle.cpp.o.d -o CMakeFiles/smoke_test_particle.dir/test/smoke_test_particle.cpp.o -c /users/smithc11/develop/pumipicDev/pumi-pic/particle_structs/test/smoke_test_particle.cpp
/users/smithc11/develop/pumipicDev/pumi-pic/support/SupportKK.h(163): error: no instance of function template "Kokkos::View<DataType, Properties...>::operator() [with DataType=Vector3 *, Properties=<Kokkos::LayoutLeft, Kokkos::CudaSpace::device_type>]" matches the argument list
argument types are: (const int)
object type is: const Kokkos::View<Vector3 *, Kokkos::LayoutLeft, Kokkos::CudaSpace::device_type>
new_view(i) = view(start + i);
^
detected during:
instantiation of "View pumipic::Subview<T>::subview(View, int, int) [with T=Vector3 *, View=Kokkos::View<Vector3 *, Kokkos::LayoutLeft, Kokkos::CudaSpace::device_type>]" at line 54 of /users/smithc11/develop/pumipicDev/pumi-pic/support/ViewComm_gpu.hpp
instantiation of "pumipic::IsGPU<pumipic::ViewSpace<ViewT>> pumipic::PS_Comm_Isend(ViewT, int, int, int, int, MPI_Comm, MPI_Request *) [with ViewT=Kokkos::View<Vector3 *, Kokkos::LayoutLeft, Kokkos::CudaSpace::device_type>]" at line 276 of /users/smithc11/develop/pumipicDev/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h
instantiation of "pumipic::SendViewsImpl<Device, T, Types...>::SendViewsImpl(pumipic::MemberTypeViews, int, int, int, int, MPI_Comm, MPI_Request *) [with Device=Kokkos::CudaSpace::device_type, T=Vector3, Types=<short, int>]" at line 277 of /users/smithc11/develop/pumipicDev/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h
instantiation of "pumipic::SendViewsImpl<Device, T, Types...>::SendViewsImpl(pumipic::MemberTypeViews, int, int, int, int, MPI_Comm, MPI_Request *) [with Device=Kokkos::CudaSpace::device_type, T=int, Types=<Vector3, short, int>]" at line 284 of /users/smithc11/develop/pumipicDev/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h
instantiation of "pumipic::SendViews<Device, pumipic::MemberTypes<Types...>>::SendViews(pumipic::MemberTypeViews, int, int, int, int, MPI_Comm, MPI_Request *) [with Device=Kokkos::CudaSpace::device_type, Types=<int, Vector3, short, int>]" at line 161 of /users/smithc11/develop/pumipicDev/pumi-pic/particle_structs/src/scs/SCS_migrate.h
instantiation of "void pumipic::SellCSigma<DataTypes, MemSpace>::migrate(pumipic::ParticleStructure<DataTypes, MemSpace>::kkLidView, pumipic::ParticleStructure<DataTypes, MemSpace>::kkLidView, pumipic::Distributor<MemSpace>, pumipic::ParticleStructure<DataTypes, MemSpace>::kkLidView, pumipic::ParticleStructure<DataTypes, MemSpace>::MTVs) [with DataTypes=Types, MemSpace=MemSpace]" at line 307 of /users/smithc11/develop/pumipicDev/pumi-pic/particle_structs/src/scs/SellCSigma.h
instantiation of "pumipic::SellCSigma<DataTypes, MemSpace>::SellCSigma(pumipic::SellCSigma<DataTypes, MemSpace>::PolicyType &, pumipic::lid_t, pumipic::lid_t, pumipic::lid_t, pumipic::lid_t, pumipic::ParticleStructure<DataTypes, MemSpace>::kkLidView, pumipic::ParticleStructure<DataTypes, MemSpace>::kkGidView, pumipic::ParticleStructure<DataTypes, MemSpace>::kkLidView, pumipic::ParticleStructure<DataTypes, MemSpace>::MTVs, MPI_Comm) [with DataTypes=Types, MemSpace=MemSpace]" at line 35 of /users/smithc11/develop/pumipicDev/pumi-pic/particle_structs/test/smoke_test_particle.cpp
Frontier
env
module load rocm #6.2.4
export DEVICE_ARCH=AMD_GFX90A
kokkos cmake configure
SOURCE_DIR=$1
BUILD_DIR=$2
ENABLE_HIP=ON
ENABLE_OPENMP=OFF
if [ "${DEVICE_ARCH}" = "NATIVE" ]; then
echo "BUILDING WITH NATIVE CPU ARCH"
ENABLE_HIP=OFF
ENABLE_OPENMP=ON
fi
cmake -S $SOURCE_DIR/kokkos -B $BUILD_DIR/${DEVICE_ARCH}/kokkos \
-DCMAKE_INSTALL_PREFIX=$BUILD_DIR/${DEVICE_ARCH}/kokkos/install \
-DCMAKE_CXX_COMPILER=hipcc \
-DCMAKE_CXX_STANDARD=17 \
-DKokkos_ARCH_${DEVICE_ARCH}=ON \
-DKokkos_ENABLE_SERIAL=ON \
-DKokkos_ENABLE_OPENMP=${ENABLE_OPENMP} \
-DKokkos_ENABLE_HIP=${ENABLE_HIP} \
-DKokkos_ENABLE_ROCTHRUST:BOOL=ON \
-DBUILD_SHARED_LIBS=ON \
-DCRAYPE_LINK_TYPE=dynamic
cmake --build $BUILD_DIR/${DEVICE_ARCH}/kokkos -j12 --target install
pumipic cmake configure
#!/bin/bash -e
SOURCE_DIR=$1
BUILD_DIR=$2
export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:\
$BUILD_DIR/${DEVICE_ARCH}/kokkos/install/lib64/cmake/Kokkos:\
$BUILD_DIR/engpar/install/lib/cmake/EnGPar
bdir=$BUILD_DIR/${DEVICE_ARCH}/pumi-pic
set -x
cmake -S $SOURCE_DIR/pumi-pic -B $bdir \
-DCMAKE_INSTALL_PREFIX=$bdir/install \
-DCMAKE_CXX_COMPILER=CC \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DIS_TESTING=OFF \
-DPS_IS_TESTING=OFF \
-DPS_USE_GPU_AWARE_MPI=ON \
-DTEST_DATA_DIR=$SOURCE_DIR/pumi-pic/pumipic-data \
-DOmega_h_PREFIX=$BUILD_DIR/${DEVICE_ARCH}/omega_h/install \
-DENABLE_CABANA=off
cmake --build $bdir -j12 --target install
set +x
verbose make output
[ 42%] Building CXX object particle_structs/CMakeFiles/smoke_test_particle.dir/test/smoke_test_particle.cpp.o
cd /lustre/orion/csc679/scratch/cwsmith/pcmsDev/builds/AMD_GFX90A/pumi-pic/particle_structs && /opt/cray/pe/craype/2.7.33/bin/CC -DKOKKOS_DEPENDENCE -DKOKKOS_ENABLED -DPP_USE_GPU -DPP_USE_HIP -DPS_GPU_AWARE_MPI -DPUMIPIC_PRINT_ENABLED -DUSE_PROF_API=1 -D__HIP_PLATFORM_AMD__=1 -I/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/support -I/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src -I/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/support -I/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/scs -I/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/csr -I/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/cabm -I/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/dps -I/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/support/support -isystem /lustre/orion/csc679/scratch/cwsmith/pcmsDev/builds/AMD_GFX90A/kokkos/install/include -O2 -g -DNDEBUG -fno-gpu-rdc --offload-arch=gfx90a -x hip --offload-arch=gfx90a -o CMakeFiles/smoke_test_particle.dir/test/smoke_test_particle.cpp.o -c /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/test/smoke_test_particle.cpp
In file included from /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/particle_structs.hpp:2:
In file included from /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/particle_structure.hpp:3:
In file included from /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/support/Segment.h:3:
In file included from /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:3:
In file included from /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/support/ViewComm.h:7:
/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/support/ppMemUsage.hpp:21:33: warning: 'mallinfo' is deprecated [-Wdeprecated-declarations]
21 | struct mallinfo meminfo_now = mallinfo();
| ^
/usr/include/malloc.h:114:48: note: 'mallinfo' has been explicitly marked deprecated here
114 | extern struct mallinfo mallinfo (void) __THROW __MALLOC_DEPRECATED;
| ^
/usr/include/malloc.h:32:30: note: expanded from macro '__MALLOC_DEPRECATED'
32 | # define __MALLOC_DEPRECATED __attribute_deprecated__
| ^
/usr/include/sys/cdefs.h:356:51: note: expanded from macro '__attribute_deprecated__'
356 | # define __attribute_deprecated__ __attribute__ ((__deprecated__))
| ^
In file included from /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/particle_structs.hpp:3:
In file included from /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/ps_for.hpp:3:
In file included from /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/particle_structs.hpp:4:
/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/scs/SellCSigma.h:410:24: warning: 'HostMirror' is deprecated: Use host_mirror_type instead. [-Wdeprecated-declarations]
410 | Kokkos::View<bool*>::HostMirror particle_mask_host = deviceToHost(particle_mask);
| ^
/lustre/orion/csc679/scratch/cwsmith/pcmsDev/builds/AMD_GFX90A/kokkos/install/include/Kokkos_View.hpp:236:20: note: 'HostMirror' has been explicitly marked deprecated here
236 | using HostMirror KOKKOS_DEPRECATED_WITH_COMMENT(
| ^
/lustre/orion/csc679/scratch/cwsmith/pcmsDev/builds/AMD_GFX90A/kokkos/install/include/Kokkos_Macros.hpp:614:51: note: expanded from macro 'KOKKOS_DEPRECATED_WITH_COMMENT'
614 | #define KOKKOS_DEPRECATED_WITH_COMMENT(comment) [[deprecated(comment)]]
| ^
In file included from /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/particle_structs.hpp:2:
In file included from /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/particle_structure.hpp:3:
In file included from /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/support/Segment.h:3:
In file included from /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:3:
In file included from /lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/support/ViewComm.h:4:
/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/support/SupportKK.h:163:9: error: no matching function for call to object of type 'const Kokkos::View<double (*)[3], Kokkos::LayoutLeft, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>>'
163 | new_view(i) = view(start + i);
| ^~~~~~~~
/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/support/SupportKK.h:162:62: note: while substituting into a lambda expression here
162 | Kokkos::parallel_for(size, KOKKOS_LAMBDA(const int& i) {
| ^
/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/support/ViewComm_gpu.hpp:54:47: note: in instantiation of function template specialization 'pumipic::Subview<double (*)[3]>::subview<Kokkos::View<double (*)[3], Kokkos::LayoutLeft, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>>>' requested here
54 | auto subview = Subview<ViewType<ViewT> >::subview(view, offset, size);
| ^
/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:276:7: note: in instantiation of function template specialization 'pumipic::PS_Comm_Isend<Kokkos::View<double (*)[3], Kokkos::LayoutLeft, Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>>>' requested here
276 | PS_Comm_Isend(v.view(), offset, size, dest, tag, comm, reqs);
| ^
/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:277:7: note: in instantiation of member function 'pumipic::SendViewsImpl<Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, double[3], short, int>::SendViewsImpl' requested here
277 | SendViewsImpl<Device, Types...>(views+1, offset, size, dest, tag + 1, comm, reqs + 1);
| ^
/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:284:7: note: in instantiation of member function 'pumipic::SendViewsImpl<Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, int, double[3], short, int>::SendViewsImpl' requested here
284 | SendViewsImpl<Device, Types...>(views, offset, size, dest, start_tag, comm, reqs);
| ^
/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/scs/SCS_migrate.h:160:9: note: in instantiation of member function 'pumipic::SendViews<Kokkos::Device<Kokkos::HIP, Kokkos::HIPSpace>, pumipic::MemberTypes<int, double[3], short, int>>::SendViews' requested here
160 | SendViews<device_type, DataTypes>(send_particle, start_index, num_send, rank, 1,
| ^
/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/src/scs/SellCSigma.h:289:34: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::migrate' requested here
289 | SellCSigma<DataTypes, MemSpace>::SellCSigma(PolicyType& p, lid_t sig, lid_t v, lid_t ne,
| ^
/lustre/orion/csc679/scratch/cwsmith/pcmsDev/pumi-pic/particle_structs/test/smoke_test_particle.cpp:35:25: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::SellCSigma' requested here
35 | PS* structure = new ps::SellCSigma<Types, MemSpace>(policy, sigma, V, num_elems, num_ptcls, ppe,
| ^
/lustre/orion/csc679/scratch/cwsmith/pcmsDev/builds/AMD_GFX90A/kokkos/install/include/View/Kokkos_BasicView.hpp:634:3: note: candidate template ignored: requirement '(sizeof...(OtherIndexTypes)) == rank()' was not satisfied [with OtherIndexTypes = <int>]
634 | operator()(OtherIndexTypes... indices) const {
| ^
2 warnings and 1 error generated when compiling for gfx90a.
make[2]: *** [particle_structs/CMakeFiles/smoke_test_particle.dir/build.make:75: particle_structs/CMakeFiles/smoke_test_particle.dir/test/smoke_test_particle.cpp.o] Error 1
make[2]: Leaving directory '/lustre/orion/csc679/scratch/cwsmith/pcmsDev/builds/AMD_GFX90A/pumi-pic'
make[1]: *** [CMakeFiles/Makefile2:232: particle_structs/CMakeFiles/smoke_test_particle.dir/all] Error 2
make[1]: Leaving directory '/lustre/orion/csc679/scratch/cwsmith/pcmsDev/builds/AMD_GFX90A/pumi-pic'
make: *** [Makefile:146: all] Error 2
Haven't tested on SCOREC yet. However, using -DKokkos_ENABLE_IMPL_VIEW_LEGACY=ON allows compilation to pass on Frontier by converting the errors to warnings. We may need to update the code to avoid the usage of deprecated features.
Interesting. Thank you. The Cabana repo also currently requires legacy views: https://github.com/ECP-copa/Cabana/pull/833