pumi-pic
pumi-pic copied to clipboard
SCOREC
Compilation issue on ALCF Aurora
While attempting to compile PUMIPic on Aurora, I encountered the following error (log file truncated to fix in the space).
Modules used:
Currently Loaded Modules:
1) gcc-runtime/12.2.0-267awrk 5) gcc/12.2.0 9) mpich/opt/4.3.0rc3 13) gmake/4.4.1
2) gmp/6.2.1-yctcuid 6) oneapi/eng-compiler/2024.07.30.002 10) libfabric/1.20.1 14) cmake/3.27.9
3) mpfr/4.2.1-fhgnwe7 7) hwloc/master-git.1793e43-level-zero 11) cray-pals/1.4.0
4) mpc/1.3.1-ygprpb4 8) yaksa/0.3-aw2kkvy 12) cray-libpals/1.4.0
One notable thing in building kokkos and omega_h is to use below macros respectively:
-DKokkos_ARCH_INTEL_PVC=ON
-DKokkos_ENABLE_SYCL=ON
-DOmega_h_USE_SYCL=ON
commit tags:
- https://github.com/kokkos/kokkos/commit/96e2663ab86e55792fbbe9b746ed16469961fe05
- https://github.com/SCOREC/omega_h/commit/eeaf6e71cadf935f59f2da23dea6d74c4bf646b8
- https://github.com/ECP-copa/Cabana/commit/896f800e37bfd7022cbb822131cfd9d1622a5823
- https://github.com/SCOREC/pumi-pic/commit/13fca71cb02dc2503697ee3d00683f6e1a7fac9e
The PUMIPic build script is below (all dependent libraries were built successfully):
module load cmake
export installroot=$PWD
export srcroot=$installroot/../
# kokkos
export kk=$installroot/kokkos/install
export kksrc=$srcroot/kokkos
# omega_h
export oh=$installroot/omega_h/install
# EnGPar
export EnGPar=$installroot/EnGPar/install
# Cabana
export cabana=$installroot/cabana/install
# pumi-pic
export pumipicsrc=$srcroot/pumi-pic
export testdir=$pumipicsrc/pumipic-data
export pumipic=$installroot/pumi-pic/install
export CMAKE_PREFIX_PATH=$kk:$oh:$EnGPar:$cabana:$CMAKE_PREFIX_PATH
cd $installroot
mkdir -p pumi-pic/build
cd pumi-pic/build
cmake $pumipicsrc -DCMAKE_BUILD_TYPE=Release \
-DBUILD_SHARED_LIBS=OFF \
-DCMAKE_CXX_COMPILER=mpicxx \
-DENABLE_CABANA=ON \
-DIS_TESTING=OFF \
-DTEST_DATA_DIR=$testdir \
-DPS_IS_TESTING=OFF \
-DCMAKE_INSTALL_PREFIX=$pumipic
make -j4 install
-- The CXX compiler identification is IntelLLVM 2025.0.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/aurora/24.180.3/spack/unified/0.8.0/install/linux-sles15-x86_64/oneapi-2024.07.30.002/mpich-4.3.0rc3-fzmrfta/bin/mpicxx - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- CMAKE_VERSION: 3.27.9
-- pumipic_VERSION: 2.1.5
-- BUILD_TESTING: OFF
-- CMAKE_INSTALL_PREFIX: /home/zhangc20/dsmc/comet/install/pumi-pic/install
-- IS_TESTING: OFF
-- pumipic_USE_MPI: ON
CMake Warning (dev) at cmake/bob.cmake:115 (find_package):
Policy CMP0074 is not set: find_package uses <PackageName>_ROOT variables.
Run "cmake --help-policy CMP0074" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.
Environment variable MPI_ROOT is set to:
/opt/aurora/24.180.3/spack/unified/0.8.0/install/linux-sles15-x86_64/oneapi-2024.07.30.002/mpich-4.3.0rc3-fzmrfta
For compatibility, CMake is ignoring the variable.
Call Stack (most recent call first):
cmake/bob.cmake:127 (bob_private_dep)
CMakeLists.txt:38 (bob_public_dep)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Found MPI_CXX: /opt/aurora/24.180.3/spack/unified/0.8.0/install/linux-sles15-x86_64/oneapi-2024.07.30.002/mpich-4.3.0rc3-fzmrfta/bin/mpicxx (found version "4.1")
-- Found MPI: TRUE (found version "4.1")
-- pumipic_USE_Omega_h: ON
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.11")
-- Enabled Kokkos devices: SERIAL;SYCL
-- Omega_h_CONFIG: /home/zhangc20/dsmc/comet/install/omega_h/install/lib64/cmake/Omega_h/Omega_hConfig.cmake
-- Omega_h_VERSION: 10.8.7
-- Looking for ask_revClass
-- Looking for ask_revClass - not found
-- Found EnGPar
-- pumipic_USE_EnGPar: ON
-- EnGPar_CONFIG: /home/zhangc20/dsmc/comet/install/EnGPar/install/lib/cmake/EnGPar/EnGParConfig.cmake
-- EnGPar_VERSION: 1.1.0
-- The C compiler identification is IntelLLVM 2025.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/aurora/24.180.3/updates/oneapi/compiler/eng-20240629/bin/icx - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- pumipic_USE_Cabana: ON
-- Enabled Kokkos devices: SERIAL;SYCL
-- Found MPI: TRUE (found version "4.1") found components: CXX
-- Cabana_CONFIG: /home/zhangc20/dsmc/comet/install/cabana/install/share/cmake/Cabana/CabanaConfig.cmake
-- Cabana_VERSION: 0.8.0-dev
-- pumipic_USE_Kokkos: ON
-- Enabled Kokkos devices: SERIAL;SYCL
-- Kokkos_CONFIG: /home/zhangc20/dsmc/comet/install/kokkos/install/lib64/cmake/Kokkos/KokkosConfig.cmake
-- Kokkos_VERSION: 4.6.99
-- CMAKE_CXX_FLAGS
-- PS_IS_TESTING: OFF
-- FP64: ON
-- FP32: OFF
-- Configuring done (2.0s)
-- Generating done (0.3s)
CMake Warning:
Manually-specified variables were not used by the project:
TEST_DATA_DIR
-- Build files have been written to: /home/zhangc20/dsmc/comet/install/pumi-pic/build
[ 21%] Building CXX object support/CMakeFiles/support.dir/ppAssert.cpp.o
[ 21%] Building CXX object support/CMakeFiles/support.dir/ViewComm.cpp.o
[ 21%] Building CXX object support/CMakeFiles/support.dir/ppTiming.cpp.o
[ 21%] Building CXX object support/CMakeFiles/support.dir/ppPrint.cpp.o
Compilation from IR - skipping loading of FCL
Compilation from IR - skipping loading of FCL
Compilation from IR - skipping loading of FCL
Build succeeded.
Build succeeded.
Build succeeded.
Compilation from IR - skipping loading of FCL
Build succeeded.
[ 26%] Linking CXX static library libsupport.a
[ 26%] Built target support
[ 31%] Building CXX object particle_structs/src/CMakeFiles/particleStructs.dir/support/prebarrier.cpp.o
Compilation from IR - skipping loading of FCL
Build succeeded.
[ 36%] Linking CXX static library libparticleStructs.a
[ 36%] Built target particleStructs
[ 57%] Building CXX object src/CMakeFiles/pumipic-core.dir/pumipic_input.cpp.o
[ 57%] Building CXX object src/CMakeFiles/pumipic-core.dir/pumipic_lb.cpp.o
[ 57%] Building CXX object particle_structs/CMakeFiles/smoke_test_particle.dir/test/smoke_test_particle.cpp.o
[ 57%] Building CXX object src/CMakeFiles/pumipic-core.dir/pumipic_part_construct.cpp.o
/home/zhangc20/dsmc/comet/pumi-pic/src/pumipic_lb.cpp:410:7: error: no matching function for call to 'PS_Comm_Irecv'
410 | PS_Comm_Irecv(elem_sbars.view(), nents_start, nents_size,
| ^~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:43:27: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with ViewT = View<int *>]
43 | IsHost<ViewSpace<ViewT> > PS_Comm_Irecv(ViewT view, int offset, int size,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/src/pumipic_lb.cpp:412:7: error: no matching function for call to 'PS_Comm_Isend'
412 | PS_Comm_Isend(elem_sbars.view(), start, nce,
| ^~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:23:27: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with ViewT = View<int *>]
23 | IsHost<ViewSpace<ViewT> > PS_Comm_Isend(ViewT view, int offset, int size,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/src/pumipic_lb.cpp:416:5: error: no matching function for call to 'PS_Comm_Waitall'
416 | PS_Comm_Waitall<Kokkos::DefaultExecutionSpace>(nbuffers, recv_requests,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:83:15: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with Space = Kokkos::DefaultExecutionSpace]
83 | IsHost<Space> PS_Comm_Waitall(int num_reqs, MPI_Request* reqs, MPI_Status* stats) {
| ^
/home/zhangc20/dsmc/comet/pumi-pic/src/pumipic_lb.cpp:419:5: error: no matching function for call to 'PS_Comm_Waitall'
419 | PS_Comm_Waitall<Kokkos::DefaultExecutionSpace>(nbuffers, send_requests,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:83:15: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with Space = Kokkos::DefaultExecutionSpace]
83 | IsHost<Space> PS_Comm_Waitall(int num_reqs, MPI_Request* reqs, MPI_Status* stats) {
| ^
Compilation from IR - skipping loading of FCL
Build succeeded.
In file included from /home/zhangc20/dsmc/comet/pumi-pic/src/pumipic_part_construct.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/src/pumipic_mesh.hpp:2:
In file included from /home/zhangc20/dsmc/comet/install/omega_h/install/include/Omega_h_mesh.hpp:4:
In file included from /home/zhangc20/dsmc/comet/install/omega_h/install/include/Omega_h_adj.hpp:4:
In file included from /home/zhangc20/dsmc/comet/install/omega_h/install/include/Omega_h_few.hpp:8:
In file included from /home/zhangc20/dsmc/comet/install/omega_h/install/include/Omega_h_defines.hpp:4:
In file included from /home/zhangc20/dsmc/comet/install/omega_h/install/include/Omega_h_macros.h:7:
In file included from /home/zhangc20/dsmc/comet/install/kokkos/install/include/Kokkos_Core.hpp:45:
In file included from /home/zhangc20/dsmc/comet/install/kokkos/install/include/KokkosCore_Config_DeclareBackend.hpp:23:
In file included from /home/zhangc20/dsmc/comet/install/kokkos/install/include/decl/Kokkos_Declare_SYCL.hpp:35:
/home/zhangc20/dsmc/comet/install/kokkos/install/include/SYCL/Kokkos_SYCL_ParallelScan_Range.hpp:203:13: error: no matching function for call to object of type 'const (anonymous namespace)::GlobalNumberer'
203 | functor(global_id + begin, local_value, false);
| ^~~~~~~
/home/zhangc20/dsmc/comet/install/kokkos/install/include/SYCL/Kokkos_SYCL_ParallelScan_Range.hpp:182:48: note: while substituting into a lambda expression here
182 | auto lambda = [=](sycl::nd_item<1> item) {
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/SYCL/Kokkos_SYCL_ParallelScan_Range.hpp:181:80: note: while substituting into a lambda expression here
181 | sycl_device_ptr<value_type> group_results_) {
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/SYCL/Kokkos_SYCL_ParallelScan_Range.hpp:384:9: note: in instantiation of function template specialization 'Kokkos::Impl::ParallelScanSYCLBase<(anonymous namespace)::GlobalNumberer, void, Kokkos::SYCL, Kokkos::Schedule<Kokkos::Static>, int>::sycl_direct_launch<Kokkos::Impl::SYCLFunctionWrapper<Kokkos::Impl::CombinedFunctorReducer<(anonymous namespace)::GlobalNumberer, Kokkos::Impl::FunctorAnalysis<Kokkos::Impl::FunctorPatternInterface::SCAN, Kokkos::RangePolicy<Kokkos::SYCL, Kokkos::Schedule<Kokkos::Static>, int>, (anonymous namespace)::GlobalNumberer, void>::Reducer>, Kokkos::Impl::SYCLInternal::USMObjectMem<sycl::usm::alloc::host>>>' requested here
384 | sycl_direct_launch(functor_wrapper, functor_wrapper.get_copy_event());
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/SYCL/Kokkos_SYCL_ParallelScan_Range.hpp:408:11: note: in instantiation of function template specialization 'Kokkos::Impl::ParallelScanSYCLBase<(anonymous namespace)::GlobalNumberer, void, Kokkos::SYCL, Kokkos::Schedule<Kokkos::Static>, int>::impl_execute<(lambda at /home/zhangc20/dsmc/comet/install/kokkos/install/include/SYCL/Kokkos_SYCL_ParallelScan_Range.hpp:408:24)>' requested here
408 | Base::impl_execute([]() {});
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/Kokkos_Parallel.hpp:364:11: note: in instantiation of member function 'Kokkos::Impl::ParallelScan<(anonymous namespace)::GlobalNumberer, Kokkos::RangePolicy<Kokkos::SYCL, Kokkos::Schedule<Kokkos::Static>, int>>::execute' requested here
364 | closure.execute();
| ^
/home/zhangc20/dsmc/comet/install/omega_h/install/include/Omega_h_scan.hpp:46:22: note: in instantiation of function template specialization 'Kokkos::parallel_scan<Kokkos::RangePolicy<Kokkos::SYCL, Kokkos::Schedule<Kokkos::Static>, int>, (anonymous namespace)::GlobalNumberer, void>' requested here
46 | if (n > 0) Kokkos::parallel_scan(name, policy(n), f);
| ^
/home/zhangc20/dsmc/comet/pumi-pic/src/pumipic_part_construct.cpp:372:14: note: in instantiation of function template specialization 'Omega_h::parallel_scan<(anonymous namespace)::GlobalNumberer>' requested here
372 | Omega_h::parallel_scan(owner.size(), gnr);
| ^
/home/zhangc20/dsmc/comet/pumi-pic/src/pumipic_part_construct.cpp:346:25: note: candidate function not viable: no known conversion from 'value_type' (aka 'int') to 'Omega_h::LO *' (aka 'int *') for 2nd argument; take the address of the argument with &
346 | OMEGA_H_DEVICE void operator()(const size_type& i, value_type vals,
| ^ ~~~~~~~~~~~~~~~
In file included from /home/zhangc20/dsmc/comet/pumi-pic/src/pumipic_part_construct.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/src/pumipic_mesh.hpp:2:
In file included from /home/zhangc20/dsmc/comet/install/omega_h/install/include/Omega_h_mesh.hpp:4:
In file included from /home/zhangc20/dsmc/comet/install/omega_h/install/include/Omega_h_adj.hpp:4:
In file included from /home/zhangc20/dsmc/comet/install/omega_h/install/include/Omega_h_few.hpp:8:
In file included from /home/zhangc20/dsmc/comet/install/omega_h/install/include/Omega_h_defines.hpp:4:
In file included from /home/zhangc20/dsmc/comet/install/omega_h/install/include/Omega_h_macros.h:7:
In file included from /home/zhangc20/dsmc/comet/install/kokkos/install/include/Kokkos_Core.hpp:45:
In file included from /home/zhangc20/dsmc/comet/install/kokkos/install/include/KokkosCore_Config_DeclareBackend.hpp:23:
In file included from /home/zhangc20/dsmc/comet/install/kokkos/install/include/decl/Kokkos_Declare_SYCL.hpp:35:
/home/zhangc20/dsmc/comet/install/kokkos/install/include/SYCL/Kokkos_SYCL_ParallelScan_Range.hpp:350:17: error: no matching function for call to object of type 'const (anonymous namespace)::GlobalNumberer'
350 | functor(global_id + begin, update, true);
| ^~~~~~~
/home/zhangc20/dsmc/comet/install/kokkos/install/include/SYCL/Kokkos_SYCL_ParallelScan_Range.hpp:335:38: note: while substituting into a lambda expression here
335 | [=](sycl::nd_item<1> item) {
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/SYCL/Kokkos_SYCL_ParallelScan_Range.hpp:322:67: note: while substituting into a lambda expression here
322 | auto update_global_results = q.submit([&](sycl::handler& cgh) {
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/SYCL/Kokkos_SYCL_ParallelScan_Range.hpp:384:9: note: in instantiation of function template specialization 'Kokkos::Impl::ParallelScanSYCLBase<(anonymous namespace)::GlobalNumberer, void, Kokkos::SYCL, Kokkos::Schedule<Kokkos::Static>, int>::sycl_direct_launch<Kokkos::Impl::SYCLFunctionWrapper<Kokkos::Impl::CombinedFunctorReducer<(anonymous namespace)::GlobalNumberer, Kokkos::Impl::FunctorAnalysis<Kokkos::Impl::FunctorPatternInterface::SCAN, Kokkos::RangePolicy<Kokkos::SYCL, Kokkos::Schedule<Kokkos::Static>, int>, (anonymous namespace)::GlobalNumberer, void>::Reducer>, Kokkos::Impl::SYCLInternal::USMObjectMem<sycl::usm::alloc::host>>>' requested here
384 | sycl_direct_launch(functor_wrapper, functor_wrapper.get_copy_event());
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/SYCL/Kokkos_SYCL_ParallelScan_Range.hpp:408:11: note: in instantiation of function template specialization 'Kokkos::Impl::ParallelScanSYCLBase<(anonymous namespace)::GlobalNumberer, void, Kokkos::SYCL, Kokkos::Schedule<Kokkos::Static>, int>::impl_execute<(lambda at /home/zhangc20/dsmc/comet/install/kokkos/install/include/SYCL/Kokkos_SYCL_ParallelScan_Range.hpp:408:24)>' requested here
408 | Base::impl_execute([]() {});
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/Kokkos_Parallel.hpp:364:11: note: in instantiation of member function 'Kokkos::Impl::ParallelScan<(anonymous namespace)::GlobalNumberer, Kokkos::RangePolicy<Kokkos::SYCL, Kokkos::Schedule<Kokkos::Static>, int>>::execute' requested here
364 | closure.execute();
| ^
/home/zhangc20/dsmc/comet/install/omega_h/install/include/Omega_h_scan.hpp:46:22: note: in instantiation of function template specialization 'Kokkos::parallel_scan<Kokkos::RangePolicy<Kokkos::SYCL, Kokkos::Schedule<Kokkos::Static>, int>, (anonymous namespace)::GlobalNumberer, void>' requested here
46 | if (n > 0) Kokkos::parallel_scan(name, policy(n), f);
| ^
/home/zhangc20/dsmc/comet/pumi-pic/src/pumipic_part_construct.cpp:372:14: note: in instantiation of function template specialization 'Omega_h::parallel_scan<(anonymous namespace)::GlobalNumberer>' requested here
372 | Omega_h::parallel_scan(owner.size(), gnr);
| ^
/home/zhangc20/dsmc/comet/pumi-pic/src/pumipic_part_construct.cpp:346:25: note: candidate function not viable: no known conversion from 'value_type' (aka 'int') to 'Omega_h::LO *' (aka 'int *') for 2nd argument; take the address of the argument with &
346 | OMEGA_H_DEVICE void operator()(const size_type& i, value_type vals,
| ^ ~~~~~~~~~~~~~~~
4 errors generated.
make[2]: *** [src/CMakeFiles/pumipic-core.dir/build.make:76: src/CMakeFiles/pumipic-core.dir/pumipic_lb.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
/home/zhangc20/dsmc/comet/pumi-pic/src/pumipic_part_construct.cpp:288:87: error: SYCL kernel cannot call a variadic function
288 | pumipic::printInfo("%d Class id is too low %d on entitiy %d\n", rank, c_id, id);
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/SYCL/Kokkos_SYCL_ParallelFor_Range.hpp:56:9: note: called by 'operator()'
56 | m_functor_wrapper.get_functor()(shifted_id);
| ^
/opt/aurora/24.180.3/updates/oneapi/compiler/eng-20240629/bin/compiler/../../include/sycl/handler.hpp:1648:5: note: called by 'kernel_parallel_for<Kokkos::Impl::FunctorWrapperRangePolicyParallelForCustom<Kokkos::Impl::SYCLFunctionWrapper<(lambda at /home/zhangc20/dsmc/comet/pumi-pic/src/pumipic_part_construct.cpp:285:32), Kokkos::Impl::SYCLInternal::USMObjectMem<sycl::usm::alloc::host>>, Kokkos::RangePolicy<Kokkos::SYCL, Kokkos::Schedule<Kokkos::Static>, int>>, sycl::nd_item<>, Kokkos::Impl::FunctorWrapperRangePolicyParallelForCustom<Kokkos::Impl::SYCLFunctionWrapper<(lambda at /home/zhangc20/dsmc/comet/pumi-pic/src/pumipic_part_construct.cpp:285:32), Kokkos::Impl::SYCLInternal::USMObjectMem<sycl::usm::alloc::host>>, Kokkos::RangePolicy<Kokkos::SYCL, Kokkos::Schedule<Kokkos::Static>, int>>>'
1648 | KernelFunc(detail::Builder::getElement(detail::declptr<ElementType>()));
| ^
/home/zhangc20/dsmc/comet/pumi-pic/src/pumipic_part_construct.cpp:290:88: error: SYCL kernel cannot call a variadic function
290 | pumipic::printInfo("%d Class id is too high %d on entitiy %d\n", rank, c_id, id);
| ^
4 errors generated.
make[2]: *** [src/CMakeFiles/pumipic-core.dir/build.make:104: src/CMakeFiles/pumipic-core.dir/pumipic_part_construct.cpp.o] Error 1
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/ps_for.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:4:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:565:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_rebuild.h:136:17: warning: 'atomic_increment<int>' is deprecated: Use atomic_inc() instead! [-Wdeprecated-declarations]
136 | Kokkos::atomic_increment<lid_t>(&(new_particles_per_elem(new_elem)));
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_rebuild.h:133:108: note: while substituting into a lambda expression here
133 | auto countNewParticles = PS_LAMBDA(const lid_t& element_id, const lid_t& particle_id, const bool& mask){
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:289:34: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::rebuild' requested here
289 | SellCSigma<DataTypes, MemSpace>::SellCSigma(PolicyType& p, lid_t sig, lid_t v, lid_t ne,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:35:25: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::SellCSigma' requested here
35 | PS* structure = new ps::SellCSigma<Types, MemSpace>(policy, sigma, V, num_elems, num_ptcls, ppe,
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/Kokkos_Atomics_Desul_Wrapper.hpp:134:19: note: 'atomic_increment<int>' has been explicitly marked deprecated here
134 | template<class T> KOKKOS_DEPRECATED_WITH_COMMENT("Use atomic_inc() instead!") KOKKOS_FUNCTION Impl::enable_if_atomic_t<T, void> atomic_increment(T* ptr) { atomic_inc(ptr); }
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/Kokkos_Macros.hpp:577:51: note: expanded from macro 'KOKKOS_DEPRECATED_WITH_COMMENT'
577 | #define KOKKOS_DEPRECATED_WITH_COMMENT(comment) [[deprecated(comment)]]
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/ps_for.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:4:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:565:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_rebuild.h:156:17: warning: 'atomic_increment<int>' is deprecated: Use atomic_inc() instead! [-Wdeprecated-declarations]
156 | Kokkos::atomic_increment<lid_t>(&(new_particles_per_elem(new_elem)));
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_rebuild.h:154:103: note: while substituting into a lambda expression here
154 | Kokkos::parallel_for("rebuild_count", new_particle_elements.size(), KOKKOS_LAMBDA(const lid_t& i) {
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:289:34: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::rebuild' requested here
289 | SellCSigma<DataTypes, MemSpace>::SellCSigma(PolicyType& p, lid_t sig, lid_t v, lid_t ne,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:35:25: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::SellCSigma' requested here
35 | PS* structure = new ps::SellCSigma<Types, MemSpace>(policy, sigma, V, num_elems, num_ptcls, ppe,
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/Kokkos_Atomics_Desul_Wrapper.hpp:134:19: note: 'atomic_increment<int>' has been explicitly marked deprecated here
134 | template<class T> KOKKOS_DEPRECATED_WITH_COMMENT("Use atomic_inc() instead!") KOKKOS_FUNCTION Impl::enable_if_atomic_t<T, void> atomic_increment(T* ptr) { atomic_inc(ptr); }
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/Kokkos_Macros.hpp:577:51: note: expanded from macro 'KOKKOS_DEPRECATED_WITH_COMMENT'
577 | #define KOKKOS_DEPRECATED_WITH_COMMENT(comment) [[deprecated(comment)]]
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/ps_for.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:4:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:565:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_rebuild.h:20:17: warning: 'atomic_increment<int>' is deprecated: Use atomic_inc() instead! [-Wdeprecated-declarations]
20 | Kokkos::atomic_increment<lid_t>(&(new_particles_per_row(new_row)));
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_rebuild.h:12:108: note: while substituting into a lambda expression here
12 | auto countNewParticles = PS_LAMBDA(const lid_t& element_id, const lid_t& particle_id, const bool& mask){
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_rebuild.h:185:25: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::reshuffle' requested here
185 | if (tryShuffling && reshuffle(new_element, new_particle_elements, new_particles)) {
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:289:34: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::rebuild' requested here
289 | SellCSigma<DataTypes, MemSpace>::SellCSigma(PolicyType& p, lid_t sig, lid_t v, lid_t ne,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:35:25: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::SellCSigma' requested here
35 | PS* structure = new ps::SellCSigma<Types, MemSpace>(policy, sigma, V, num_elems, num_ptcls, ppe,
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/Kokkos_Atomics_Desul_Wrapper.hpp:134:19: note: 'atomic_increment<int>' has been explicitly marked deprecated here
134 | template<class T> KOKKOS_DEPRECATED_WITH_COMMENT("Use atomic_inc() instead!") KOKKOS_FUNCTION Impl::enable_if_atomic_t<T, void> atomic_increment(T* ptr) { atomic_inc(ptr); }
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/Kokkos_Macros.hpp:577:51: note: expanded from macro 'KOKKOS_DEPRECATED_WITH_COMMENT'
577 | #define KOKKOS_DEPRECATED_WITH_COMMENT(comment) [[deprecated(comment)]]
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/ps_for.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:4:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:565:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_rebuild.h:24:17: warning: 'atomic_increment<int>' is deprecated: Use atomic_inc() instead! [-Wdeprecated-declarations]
24 | Kokkos::atomic_increment<lid_t>(&(num_holes_per_row(row)));
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/Kokkos_Atomics_Desul_Wrapper.hpp:134:19: note: 'atomic_increment<int>' has been explicitly marked deprecated here
134 | template<class T> KOKKOS_DEPRECATED_WITH_COMMENT("Use atomic_inc() instead!") KOKKOS_FUNCTION Impl::enable_if_atomic_t<T, void> atomic_increment(T* ptr) { atomic_inc(ptr); }
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/Kokkos_Macros.hpp:577:51: note: expanded from macro 'KOKKOS_DEPRECATED_WITH_COMMENT'
577 | #define KOKKOS_DEPRECATED_WITH_COMMENT(comment) [[deprecated(comment)]]
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/ps_for.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:4:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:565:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_rebuild.h:31:17: warning: 'atomic_increment<int>' is deprecated: Use atomic_inc() instead! [-Wdeprecated-declarations]
31 | Kokkos::atomic_increment<lid_t>(&(new_particles_per_row(new_row)));
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_rebuild.h:28:105: note: while substituting into a lambda expression here
28 | Kokkos::parallel_for("reshuffle_count", new_particle_elements.size(), KOKKOS_LAMBDA(const lid_t& i) {
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_rebuild.h:185:25: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::reshuffle' requested here
185 | if (tryShuffling && reshuffle(new_element, new_particle_elements, new_particles)) {
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:289:34: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::rebuild' requested here
289 | SellCSigma<DataTypes, MemSpace>::SellCSigma(PolicyType& p, lid_t sig, lid_t v, lid_t ne,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:35:25: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::SellCSigma' requested here
35 | PS* structure = new ps::SellCSigma<Types, MemSpace>(policy, sigma, V, num_elems, num_ptcls, ppe,
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/Kokkos_Atomics_Desul_Wrapper.hpp:134:19: note: 'atomic_increment<int>' has been explicitly marked deprecated here
134 | template<class T> KOKKOS_DEPRECATED_WITH_COMMENT("Use atomic_inc() instead!") KOKKOS_FUNCTION Impl::enable_if_atomic_t<T, void> atomic_increment(T* ptr) { atomic_inc(ptr); }
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/Kokkos_Macros.hpp:577:51: note: expanded from macro 'KOKKOS_DEPRECATED_WITH_COMMENT'
577 | #define KOKKOS_DEPRECATED_WITH_COMMENT(comment) [[deprecated(comment)]]
| ^
make[1]: *** [CMakeFiles/Makefile2:285: src/CMakeFiles/pumipic-core.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/ps_for.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:4:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:566:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_migrate.h:34:17: warning: 'atomic_increment<int>' is deprecated: Use atomic_inc() instead! [-Wdeprecated-declarations]
34 | Kokkos::atomic_increment(&num_send_particles(process_index));
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_migrate.h:30:115: note: while substituting into a lambda expression here
30 | auto count_sending_particles = PS_LAMBDA(const lid_t& element_id, const lid_t& particle_id, const bool& mask) {
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:289:34: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::migrate' requested here
289 | SellCSigma<DataTypes, MemSpace>::SellCSigma(PolicyType& p, lid_t sig, lid_t v, lid_t ne,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:35:25: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::SellCSigma' requested here
35 | PS* structure = new ps::SellCSigma<Types, MemSpace>(policy, sigma, V, num_elems, num_ptcls, ppe,
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/Kokkos_Atomics_Desul_Wrapper.hpp:134:19: note: 'atomic_increment<int>' has been explicitly marked deprecated here
134 | template<class T> KOKKOS_DEPRECATED_WITH_COMMENT("Use atomic_inc() instead!") KOKKOS_FUNCTION Impl::enable_if_atomic_t<T, void> atomic_increment(T* ptr) { atomic_inc(ptr); }
| ^
/home/zhangc20/dsmc/comet/install/kokkos/install/include/Kokkos_Macros.hpp:577:51: note: expanded from macro 'KOKKOS_DEPRECATED_WITH_COMMENT'
577 | #define KOKKOS_DEPRECATED_WITH_COMMENT(comment) [[deprecated(comment)]]
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/ps_for.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:4:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:566:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_migrate.h:48:7: error: no matching function for call to 'PS_Comm_Ialltoall'
48 | PS_Comm_Ialltoall(num_send_particles, 1, num_recv_particles, 1, dist.mpi_comm(), count_recv_requests);
| ^~~~~~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:289:34: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::migrate' requested here
289 | SellCSigma<DataTypes, MemSpace>::SellCSigma(PolicyType& p, lid_t sig, lid_t v, lid_t ne,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:35:25: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::SellCSigma' requested here
35 | PS* structure = new ps::SellCSigma<Types, MemSpace>(policy, sigma, V, num_elems, num_ptcls, ppe,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:108:27: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with ViewT = kkLidView]
108 | IsHost<ViewSpace<ViewT> > PS_Comm_Ialltoall(ViewT send, int send_size,
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/ps_for.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:4:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:566:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_migrate.h:55:11: error: no matching function for call to 'PS_Comm_Isend'
55 | PS_Comm_Isend(num_send_particles, i, 1, rank, 0, dist.mpi_comm(),
| ^~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:23:27: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with ViewT = kkLidView]
23 | IsHost<ViewSpace<ViewT> > PS_Comm_Isend(ViewT view, int offset, int size,
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/ps_for.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:4:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:566:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_migrate.h:57:11: error: no matching function for call to 'PS_Comm_Irecv'
57 | PS_Comm_Irecv(num_recv_particles, i, 1, rank, 0, dist.mpi_comm(),
| ^~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:43:27: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with ViewT = kkLidView]
43 | IsHost<ViewSpace<ViewT> > PS_Comm_Irecv(ViewT view, int offset, int size,
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/ps_for.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:4:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:566:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_migrate.h:64:5: error: no matching function for call to 'PS_Comm_Waitall'
64 | PS_Comm_Waitall<device_type>(num_recv_ranks, count_recv_requests, MPI_STATUSES_IGNORE);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:83:15: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with Space = device_type]
83 | IsHost<Space> PS_Comm_Waitall(int num_reqs, MPI_Request* reqs, MPI_Status* stats) {
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/ps_for.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:4:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:566:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_migrate.h:113:7: error: no matching function for call to 'PS_Comm_Waitall'
113 | PS_Comm_Waitall<device_type>(num_send_ranks, count_send_requests,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:83:15: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with Space = device_type]
83 | IsHost<Space> PS_Comm_Waitall(int num_reqs, MPI_Request* reqs, MPI_Status* stats) {
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/ps_for.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:4:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:566:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_migrate.h:157:9: error: no matching function for call to 'PS_Comm_Isend'
157 | PS_Comm_Isend(send_element, start_index, num_send, rank, 0, dist.mpi_comm(),
| ^~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:23:27: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with ViewT = kkLidView]
23 | IsHost<ViewSpace<ViewT> > PS_Comm_Isend(ViewT view, int offset, int size,
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/ps_for.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:4:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:566:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_migrate.h:168:9: error: no matching function for call to 'PS_Comm_Irecv'
168 | PS_Comm_Irecv(recv_element, start_index, num_recv, rank, 0, dist.mpi_comm(),
| ^~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:43:27: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with ViewT = kkLidView]
43 | IsHost<ViewSpace<ViewT> > PS_Comm_Irecv(ViewT view, int offset, int size,
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/ps_for.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:4:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:566:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_migrate.h:177:5: error: no matching function for call to 'PS_Comm_Waitall'
177 | PS_Comm_Waitall<device_type>(num_recvs, recv_requests, MPI_STATUSES_IGNORE);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:83:15: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with Space = device_type]
83 | IsHost<Space> PS_Comm_Waitall(int num_reqs, MPI_Request* reqs, MPI_Status* stats) {
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/ps_for.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:4:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:566:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_migrate.h:213:5: error: no matching function for call to 'PS_Comm_Waitall'
213 | PS_Comm_Waitall<device_type>(num_sends, send_requests, MPI_STATUSES_IGNORE);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:83:15: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with Space = device_type]
83 | IsHost<Space> PS_Comm_Waitall(int num_reqs, MPI_Request* reqs, MPI_Status* stats) {
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:2:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structure.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/Segment.h:3:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:276:7: error: no matching function for call to 'PS_Comm_Isend'
276 | PS_Comm_Isend(v.view(), offset, size, dest, tag, comm, reqs);
| ^~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:284:7: note: in instantiation of member function 'pumipic::SendViewsImpl<Kokkos::Device<Kokkos::SYCL, Kokkos::SYCLDeviceUSMSpace>, int, double[3], short, int>::SendViewsImpl' requested here
284 | SendViewsImpl<Device, Types...>(views, offset, size, dest, start_tag, comm, reqs);
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_migrate.h:160:9: note: in instantiation of member function 'pumipic::SendViews<Kokkos::Device<Kokkos::SYCL, Kokkos::SYCLDeviceUSMSpace>, pumipic::MemberTypes<int, double[3], short, int>>::SendViews' requested here
160 | SendViews<device_type, DataTypes>(send_particle, start_index, num_send, rank, 1,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:289:34: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::migrate' requested here
289 | SellCSigma<DataTypes, MemSpace>::SellCSigma(PolicyType& p, lid_t sig, lid_t v, lid_t ne,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:35:25: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::SellCSigma' requested here
35 | PS* structure = new ps::SellCSigma<Types, MemSpace>(policy, sigma, V, num_elems, num_ptcls, ppe,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:23:27: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with ViewT = KView]
23 | IsHost<ViewSpace<ViewT> > PS_Comm_Isend(ViewT view, int offset, int size,
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:2:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structure.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/Segment.h:3:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:276:7: error: no matching function for call to 'PS_Comm_Isend'
276 | PS_Comm_Isend(v.view(), offset, size, dest, tag, comm, reqs);
| ^~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:277:7: note: in instantiation of member function 'pumipic::SendViewsImpl<Kokkos::Device<Kokkos::SYCL, Kokkos::SYCLDeviceUSMSpace>, double[3], short, int>::SendViewsImpl' requested here
277 | SendViewsImpl<Device, Types...>(views+1, offset, size, dest, tag + 1, comm, reqs + 1);
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:284:7: note: in instantiation of member function 'pumipic::SendViewsImpl<Kokkos::Device<Kokkos::SYCL, Kokkos::SYCLDeviceUSMSpace>, int, double[3], short, int>::SendViewsImpl' requested here
284 | SendViewsImpl<Device, Types...>(views, offset, size, dest, start_tag, comm, reqs);
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_migrate.h:160:9: note: in instantiation of member function 'pumipic::SendViews<Kokkos::Device<Kokkos::SYCL, Kokkos::SYCLDeviceUSMSpace>, pumipic::MemberTypes<int, double[3], short, int>>::SendViews' requested here
160 | SendViews<device_type, DataTypes>(send_particle, start_index, num_send, rank, 1,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:289:34: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::migrate' requested here
289 | SellCSigma<DataTypes, MemSpace>::SellCSigma(PolicyType& p, lid_t sig, lid_t v, lid_t ne,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:35:25: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::SellCSigma' requested here
35 | PS* structure = new ps::SellCSigma<Types, MemSpace>(policy, sigma, V, num_elems, num_ptcls, ppe,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:23:27: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with ViewT = KView]
23 | IsHost<ViewSpace<ViewT> > PS_Comm_Isend(ViewT view, int offset, int size,
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:2:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structure.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/Segment.h:3:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:276:7: error: no matching function for call to 'PS_Comm_Isend'
276 | PS_Comm_Isend(v.view(), offset, size, dest, tag, comm, reqs);
| ^~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:277:7: note: in instantiation of member function 'pumipic::SendViewsImpl<Kokkos::Device<Kokkos::SYCL, Kokkos::SYCLDeviceUSMSpace>, short, int>::SendViewsImpl' requested here
277 | SendViewsImpl<Device, Types...>(views+1, offset, size, dest, tag + 1, comm, reqs + 1);
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:277:7: note: in instantiation of member function 'pumipic::SendViewsImpl<Kokkos::Device<Kokkos::SYCL, Kokkos::SYCLDeviceUSMSpace>, double[3], short, int>::SendViewsImpl' requested here
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:284:7: note: in instantiation of member function 'pumipic::SendViewsImpl<Kokkos::Device<Kokkos::SYCL, Kokkos::SYCLDeviceUSMSpace>, int, double[3], short, int>::SendViewsImpl' requested here
284 | SendViewsImpl<Device, Types...>(views, offset, size, dest, start_tag, comm, reqs);
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_migrate.h:160:9: note: in instantiation of member function 'pumipic::SendViews<Kokkos::Device<Kokkos::SYCL, Kokkos::SYCLDeviceUSMSpace>, pumipic::MemberTypes<int, double[3], short, int>>::SendViews' requested here
160 | SendViews<device_type, DataTypes>(send_particle, start_index, num_send, rank, 1,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:289:34: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::migrate' requested here
289 | SellCSigma<DataTypes, MemSpace>::SellCSigma(PolicyType& p, lid_t sig, lid_t v, lid_t ne,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:35:25: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::SellCSigma' requested here
35 | PS* structure = new ps::SellCSigma<Types, MemSpace>(policy, sigma, V, num_elems, num_ptcls, ppe,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:23:27: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with ViewT = KView]
23 | IsHost<ViewSpace<ViewT> > PS_Comm_Isend(ViewT view, int offset, int size,
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:2:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structure.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/Segment.h:3:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:276:7: error: no matching function for call to 'PS_Comm_Isend'
276 | PS_Comm_Isend(v.view(), offset, size, dest, tag, comm, reqs);
| ^~~~~~~~~~~~~
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:277:7: note: in instantiation of member function 'pumipic::SendViewsImpl<Kokkos::Device<Kokkos::SYCL, Kokkos::SYCLDeviceUSMSpace>, int>::SendViewsImpl' requested here
277 | SendViewsImpl<Device, Types...>(views+1, offset, size, dest, tag + 1, comm, reqs + 1);
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:277:7: note: in instantiation of member function 'pumipic::SendViewsImpl<Kokkos::Device<Kokkos::SYCL, Kokkos::SYCLDeviceUSMSpace>, short, int>::SendViewsImpl' requested here
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:277:7: note: in instantiation of member function 'pumipic::SendViewsImpl<Kokkos::Device<Kokkos::SYCL, Kokkos::SYCLDeviceUSMSpace>, double[3], short, int>::SendViewsImpl' requested here
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:284:7: note: in instantiation of member function 'pumipic::SendViewsImpl<Kokkos::Device<Kokkos::SYCL, Kokkos::SYCLDeviceUSMSpace>, int, double[3], short, int>::SendViewsImpl' requested here
284 | SendViewsImpl<Device, Types...>(views, offset, size, dest, start_tag, comm, reqs);
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SCS_migrate.h:160:9: note: in instantiation of member function 'pumipic::SendViews<Kokkos::Device<Kokkos::SYCL, Kokkos::SYCLDeviceUSMSpace>, pumipic::MemberTypes<int, double[3], short, int>>::SendViews' requested here
160 | SendViews<device_type, DataTypes>(send_particle, start_index, num_send, rank, 1,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/scs/SellCSigma.h:289:34: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::migrate' requested here
289 | SellCSigma<DataTypes, MemSpace>::SellCSigma(PolicyType& p, lid_t sig, lid_t v, lid_t ne,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:35:25: note: in instantiation of member function 'pumipic::SellCSigma<pumipic::MemberTypes<int, double[3], short, int>>::SellCSigma' requested here
35 | PS* structure = new ps::SellCSigma<Types, MemSpace>(policy, sigma, V, num_elems, num_ptcls, ppe,
| ^
/home/zhangc20/dsmc/comet/pumi-pic/support/ViewComm_host.hpp:23:27: note: candidate template ignored: requirement 'Kokkos::SpaceAccessibility<Kokkos::SYCLDeviceUSMSpace, Kokkos::HostSpace>::accessible' was not satisfied [with ViewT = KView]
23 | IsHost<ViewSpace<ViewT> > PS_Comm_Isend(ViewT view, int offset, int size,
| ^
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/test/smoke_test_particle.cpp:1:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structs.hpp:2:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/particle_structure.hpp:3:
In file included from /home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/Segment.h:3:
/home/zhangc20/dsmc/comet/pumi-pic/particle_structs/src/support/MemberTypeLibraries.h:297:7: error: no matching function for call to 'PS_Comm_Irecv'
297 | PS_Comm_Irecv(v.view(), offset, size, dest, tag, comm, reqs);
| ^~~~~~~~~~~~~ ^
@Angelyr it would be helpful if you could take a look at this. If you don't currently have a working Aurora account Chonglin can get you added to his allocation.
I had one at some point, but it seems that it got deactivated. I submitted a reactivation request.
- With @Angelyr Angel's new branch, I was able to build
pumi-picsucsessfully on Aurora. - However, I encountered the following linking error when building my code, Comet:
[ 95%] Linking CXX executable Comet
/opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/gcc-13.3.0/cmake-3.30.5-igar7ts/bin/cmake -E cmake_link_script CMakeFiles/Comet.dir/link.txt --verbose=1
/opt/aurora/24.347.0/spack/unified/0.9.2/install/linux-sles15-x86_64/oneapi-2025.0.5/mpich-develop-git.6037a7a-sxnhr7p/bin/mpicxx -g -g -Rno-debug-disables-optimization -DKOKKOS_DEPENDENCE -fsycl -fno-sycl-id-queries-fit-in-int -fsycl-dead-args-optimization -DDESUL_SYCL_DEVICE_GLOBAL_SUPPORTED CMakeFiles/Comet.dir/src/comet.cpp.o -o Comet src/libcomet.a /home/zhangc20/dsmc/comet/install/pumi-pic/install/lib/libpumipic-core.a /home/zhangc20/dsmc/comet/install/pumi-pic/install/lib/libparticleStructs.a /home/zhangc20/dsmc/comet/install/pumi-pic/install/lib/libsupport.a /home/zhangc20/dsmc/comet/install/omega_h/install/lib64/libomega_h.a /home/zhangc20/dsmc/comet/install/kokkos/install/lib64/libkokkoscontainers.a /home/zhangc20/dsmc/comet/install/kokkos/install/lib64/libkokkoscore.a -ldl /home/zhangc20/dsmc/comet/install/kokkos/install/lib64/libkokkossimd.a /usr/lib64/libz.so /home/zhangc20/dsmc/comet/install/EnGPar/install/lib/libengpar.a /home/zhangc20/dsmc/comet/install/EnGPar/install/lib/libdiffusive.a /home/zhangc20/dsmc/comet/install/EnGPar/install/lib/libmultilevel.a /home/zhangc20/dsmc/comet/install/EnGPar/install/lib/libengpar_metrics.a /home/zhangc20/dsmc/comet/install/EnGPar/install/lib/libcoloring.a /home/zhangc20/dsmc/comet/install/EnGPar/install/lib/libagi.a /home/zhangc20/dsmc/comet/install/EnGPar/install/lib/libengpar_support.a /home/zhangc20/dsmc/comet/install/EnGPar/install/lib/libpcu.a
/usr/bin/ld: src/libcomet.a(comet_init_objects.cpp.o):/home/zhangc20/dsmc/comet/install/pumi-pic/install/include/SCSPair.h:33: multiple definition of `Kokkos::ma'; /tmp/icpx-24ed47c668/comet-0e00a7.o:/home/zhangc20/dsmc/comet/install/pumi-pic/install/include/SCSPair.h:33: first defined here
/usr/bin/ld: src/libcomet.a(comet_init_objects.cpp.o):/home/zhangc20/dsmc/comet/install/pumi-pic/install/include/SCSPair.h:34: multiple definition of `Kokkos::mi'; /tmp/icpx-24ed47c668/comet-0e00a7.o:/home/zhangc20/dsmc/comet/install/pumi-pic/install/include/SCSPair.h:34: first defined here
/usr/bin/ld: src/libcomet.a(comet_input.cpp.o):/home/zhangc20/dsmc/comet/install/pumi-pic/install/include/SCSPair.h:33: multiple definition of `Kokkos::ma'; /tmp/icpx-24ed47c668/comet-0e00a7.o:/home/zhangc20/dsmc/comet/install/pumi-pic/install/include/SCSPair.h:33: first defined here
/usr/bin/ld: src/libcomet.a(comet_input.cpp.o):/home/zhangc20/dsmc/comet/install/pumi-pic/install/include/SCSPair.h:34: multiple definition of `Kokkos::mi'; /tmp/icpx-24ed47c668/comet-0e00a7.o:/home/zhangc20/dsmc/comet/install/pumi-pic/install/include/SCSPair.h:34: first defined here
- Note I also need to make a minor change from
std::maxtoKokkos::Maxfor onepumi-picfunction.
@Angelyr: after deleting the following code segment, I was able to build and link successfully. This code segment does not seem to be used anywhere, could you confirm? Thanks!
#ifdef PP_USE_GPU
namespace Kokkos {
using pumipic::MyPair;
PP_DEVICE_VAR MyPair ma = MyPair(10000000);
PP_DEVICE_VAR MyPair mi = MyPair(0);
template <>
struct reduction_identity<MyPair> {
KOKKOS_FORCEINLINE_FUNCTION constexpr static const MyPair& max() {return ma;}
KOKKOS_FORCEINLINE_FUNCTION constexpr static const MyPair& min() {return mi;}
};
}
#endif
I managed to run a single pumi-pic test: https://github.com/SCOREC/pumi-pic/blob/5900df922f2dd5df76846ff4cb7c4c14d884c961/test/testing.cmake#L98-L100
The test failed with the following log file:
NUM_OF_NODES= 1 TOTAL_NUM_RANKS= 1 RANKS_PER_NODE= 1 THREADS_PER_RANK= 1
kokkos timer reports 1ms as 0.001055 seconds
number of elements 6567 number of particles 200
particle_structs floating point value size (bits): 8
omega_h floating point value size (bits): 8
Kokkos execution space memory N6Kokkos18SYCLDeviceUSMSpaceE name N6Kokkos4SYCLE
Kokkos host execution space N6Kokkos9HostSpaceE name N6Kokkos6SerialE
reading gmsh mesh pisces/gitr.msh
Mesh loaded with <v e f r> 1549 9075 14095 6567
mesh elements classified on model face 5: 112
num ptcls per elm 1
Segmentation fault from GPU at 0x7ffc43b56000, ctx_id: 1 (CCS) type: 0 (NotPresent), level: 3 (PML4), access: 0 (Read), banned: 1, aborting.
Abort was called at 274 line in file:
/home/ubit/rpmbuild/BUILD/intel-compute-runtime-25.05.32567.18/shared/source/os_interface/linux/drm_neo.cpp
Segmentation fault from GPU at 0x7ffc43b56000, ctx_id: 1 (CCS) type: 0 (NotPresent), level: 3 (PML4), access: 0 (Read), banned: 1, aborting.
x4305c3s6b0n0.hsn.cm.aurora.alcf.anl.gov: rank 0 died from signal 6 and dumped core
Note:
- This is done by commenting out
printfin the test file and commenting out other tests to compile the code. - The error message above is similar to what I saw running Comet, which is not shown here.
I suspect there may be issues related to MPI misconfiguration. I tested a minimal MPI program, and it failed in the current environment. I'll continue troubleshooting, but let me know if anyone else has some thoughts.
Update: It seems we didn’t handle the copy and free operations for SYCL here. Fixing that passes most of the tests on my side, though a few failures remain to be resolved.
@Sichao25: Thanks for the update Sichao. I remember seeing SYCL/kokkos deep copy function calls from the stack trace of the core dump.
Would it make sense to be using Kokkos::malloc for the allocations? It seems like we are replicating that.
https://kokkos.org/kokkos-core-wiki/API/core/c_style_memory_management/malloc.html#_CPPv4I0E13kokkos_mallocPvRK6string6size_t
- As of commit 091f7df, Comet is running on Aurora and producing correct simulation results;
- Some of the log file output are out of order, for example:
pumipic buffer method BFS
pumipic safe method BFS
Above output usually appear at the beginning of the simulation when particle structure is being constructed. On Aurora, it appears at the end of the log file when simulation finishes, and after timing output.
What version of Kokkos is being used for the aurora build? I'm hitting errors with Kokkos >= 4.7.00 and master or yus/aurora-support @ 091f7df on a scorec workstation and on frontier.
- I am using
kokkos 4.6.02on Aurora. - On
CentOS 9system, I am usingkokkos 4.5.01. Both are working fine eithermasterbranch oryus/aurora-supportbranch @ 091f7df.
Just put this here in case I forgot it later. With some more tests, here are the observations:
- When using
SCSparticle structure, the code crashed at the beginning of the simulation. Sincechunk_widthsis present in the log file, I suspect it crashed duringSCSparticle structure construction.
NUM_OF_NODES= 1 TOTAL_NUM_RANKS= 12 RANKS_PER_NODE= 12 THREADS_PER_RANK= 1
Successfully created directory for writing field results on rank 0
Successfully created directory for writing field results on rank 0
Successfully created directory for writing field results on rank 0
Successfully created directory for writing field results on rank 0
terminate called after throwing an instance of 'std::runtime_error'
terminate called after throwing an instance of 'std::runtime_error'
terminate called after throwing an instance of 'std::runtime_error'
terminate called after throwing an instance of 'std::runtime_error'
terminate called after throwing an instance of 'std::runtime_error'
terminate called after throwing an instance of 'std::runtime_error'
terminate called after throwing an instance of 'std::runtime_error'
terminate called after throwing an instance of 'std::runtime_error'
terminate called after throwing an instance of 'std::runtime_error'
terminate called after throwing an instance of 'std::runtime_error'
terminate called after throwing an instance of 'std::runtime_error'
terminate called after throwing an instance of 'std::runtime_error'
what(): Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
what(): Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
what(): Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
what(): Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
what(): Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
what(): Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
what(): Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
what(): Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
what(): Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
what(): Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
what(): Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
what(): Kokkos ERROR: SYCLDeviceUSM memory space failed to allocate 1.678e+07 TiB (label="chunk_widths").
x4311c0s3b0n0.hsn.cm.aurora.alcf.anl.gov: rank 2 died from signal 6 and dumped core
- when using
CabMorDPSparticle structure, the code is running fine if there are less than~20 million particles per GPU, but crashed if each GPU has more than~20 millionparticles:
Segmentation fault from GPU at 0xff00003aa2a20000, ctx_id: 1 (CCS) type: 0 (NotPresent), level: 2 (PDP), access: 1 (Write), banned: 1, aborting.
Abort was called at 274 line in file:
/home/ubit/rpmbuild/BUILD/intel-compute-runtime-25.05.32567.18/shared/source/os_interface/linux/drm_neo.cpp
Segmentation fault from GPU at 0xff00003aa2a20000, ctx_id: 1 (CCS) type: 0 (NotPresent), level: 2 (PDP), access: 1 (Write), banned: 1, aborting.
x4502c7s4b0n0.hsn.cm.aurora.alcf.anl.gov: rank 6 died from signal 6 and dumped core
x4502c7s4b0n0.hsn.cm.aurora.alcf.anl.gov: rank 1 died from signal 15
@zhangchonglin can we start a new issue for this? Can you post which software versions you are using?