GEOS
GEOS copied to clipboard
Published
20 hours ago •
GEOS-DEV
GEOS-DEV
[Bug] Tests are not running on Pangea3
Describe the bug A clear and concise description of what the bug is.
To Reproduce Steps to reproduce the behavior:
- Build GEOSX on Pangea3 with this env
export MODULEPATH=/workrd/SCR/NUM/geosx_num/module_files:$MODULEPATH
module purge
module load cmake/3.21.4 gcc/8.4.1 cuda/11.0.3 ompi/4.1.2 openblas/0.3.18 python4geosx/p3/gcc8.4.1-ompi4.1.2
- run make test
Try to provide minimal test cases where possible to help isolate the problem.
Expected behavior All test pass
Screenshots The following tests are failing on pangea3 :
- [x] 60 - testMath (Failed)
- [ ] 107 - testDruckerPrager (Failed)
- [ ] 109 - testModifiedCamClay (Failed)
- [x] 119 - testLinearAlgebra_KrylovSolvers (Timeout)
- [ ] 149 - testTriaxial_druckerPragerExtended (Failed)
- [ ] 152 - testTriaxial_modifiedCamClay (Failed)
- [ ] 153 - testTriaxial_delftEggUseLinear (Failed)
- [ ] 154 - testTriaxial_modifiedCamClayVolumetric (Failed)
- [ ] 155 - testTriaxial_delftEggLoadPathDryUseLinear (Failed)
- [ ] 156 - testTriaxial_delftEggLoadPathWetUseLinear (Failed)
- [ ] 157 - testTriaxial_delftEggCase1 (Failed)
- [ ] 158 - testTriaxial_delftEggCase2 (Failed)
Platform (please complete the following information):
- Machine TOTAL/Pangea3
- Compiler: gcc 8.4.1
- GEOSX Version Latest
Additional context These tests were failing since a while but never tested/reported before... Filled a TPL issue by error : https://github.com/GEOSX/thirdPartyLibs/issues/198
https://github.com/GEOSX/LvArray/pull/265 fixing / https://github.com/GEOSX/GEOSX/pull/2030 test 60
Except the testMath that I fixed I don't understand what that tests do. If anyone can help...
Output of ctest --rerun-failed --output-on-failure
Test project /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo
Start 60: testMath
1/11 Test GEOS-DEV/thirdPartyLibs#60: testMath ....................................***Failed 2.86 sec
[==========] Running 102 tests from 13 test suites.
[----------] Global test environment set-up.
[----------] 8 tests from TestMath/0, where TypeParam = std::pair<int, RAJA::policy::loop::loop_exec>
[ RUN ] TestMath/0.numValues
[ OK ] TestMath/0.numValues (0 ms)
[ RUN ] TestMath/0.convert
[ OK ] TestMath/0.convert (190 ms)
[ RUN ] TestMath/0.maxAndMin
[ OK ] TestMath/0.maxAndMin (0 ms)
[ RUN ] TestMath/0.abs
[ OK ] TestMath/0.abs (0 ms)
[ RUN ] TestMath/0.square
[ OK ] TestMath/0.square (0 ms)
[ RUN ] TestMath/0.sqrtAndInvSqrt
[ OK ] TestMath/0.sqrtAndInvSqrt (0 ms)
[ RUN ] TestMath/0.trig
[ OK ] TestMath/0.trig (0 ms)
[ RUN ] TestMath/0.exponential
[ OK ] TestMath/0.exponential (0 ms)
[----------] 8 tests from TestMath/0 (190 ms total)
[----------] 8 tests from TestMath/1, where TypeParam = std::pair<long, RAJA::policy::loop::loop_exec>
[ RUN ] TestMath/1.numValues
[ OK ] TestMath/1.numValues (0 ms)
[ RUN ] TestMath/1.convert
[ OK ] TestMath/1.convert (0 ms)
[ RUN ] TestMath/1.maxAndMin
[ OK ] TestMath/1.maxAndMin (0 ms)
[ RUN ] TestMath/1.abs
[ OK ] TestMath/1.abs (0 ms)
[ RUN ] TestMath/1.square
[ OK ] TestMath/1.square (0 ms)
[ RUN ] TestMath/1.sqrtAndInvSqrt
[ OK ] TestMath/1.sqrtAndInvSqrt (0 ms)
[ RUN ] TestMath/1.trig
[ OK ] TestMath/1.trig (0 ms)
[ RUN ] TestMath/1.exponential
[ OK ] TestMath/1.exponential (0 ms)
[----------] 8 tests from TestMath/1 (0 ms total)
[----------] 8 tests from TestMath/2, where TypeParam = std::pair<long long, RAJA::policy::loop::loop_exec>
[ RUN ] TestMath/2.numValues
[ OK ] TestMath/2.numValues (0 ms)
[ RUN ] TestMath/2.convert
[ OK ] TestMath/2.convert (0 ms)
[ RUN ] TestMath/2.maxAndMin
[ OK ] TestMath/2.maxAndMin (0 ms)
[ RUN ] TestMath/2.abs
[ OK ] TestMath/2.abs (0 ms)
[ RUN ] TestMath/2.square
[ OK ] TestMath/2.square (0 ms)
[ RUN ] TestMath/2.sqrtAndInvSqrt
[ OK ] TestMath/2.sqrtAndInvSqrt (0 ms)
[ RUN ] TestMath/2.trig
[ OK ] TestMath/2.trig (0 ms)
[ RUN ] TestMath/2.exponential
[ OK ] TestMath/2.exponential (0 ms)
[----------] 8 tests from TestMath/2 (0 ms total)
[----------] 8 tests from TestMath/3, where TypeParam = std::pair<float, RAJA::policy::loop::loop_exec>
[ RUN ] TestMath/3.numValues
[ OK ] TestMath/3.numValues (0 ms)
[ RUN ] TestMath/3.convert
[ OK ] TestMath/3.convert (0 ms)
[ RUN ] TestMath/3.maxAndMin
[ OK ] TestMath/3.maxAndMin (0 ms)
[ RUN ] TestMath/3.abs
[ OK ] TestMath/3.abs (0 ms)
[ RUN ] TestMath/3.square
[ OK ] TestMath/3.square (0 ms)
[ RUN ] TestMath/3.sqrtAndInvSqrt
[ OK ] TestMath/3.sqrtAndInvSqrt (0 ms)
[ RUN ] TestMath/3.trig
[ OK ] TestMath/3.trig (0 ms)
[ RUN ] TestMath/3.exponential
[ OK ] TestMath/3.exponential (0 ms)
[----------] 8 tests from TestMath/3 (0 ms total)
[----------] 8 tests from TestMath/4, where TypeParam = std::pair<double, RAJA::policy::loop::loop_exec>
[ RUN ] TestMath/4.numValues
[ OK ] TestMath/4.numValues (0 ms)
[ RUN ] TestMath/4.convert
[ OK ] TestMath/4.convert (0 ms)
[ RUN ] TestMath/4.maxAndMin
[ OK ] TestMath/4.maxAndMin (0 ms)
[ RUN ] TestMath/4.abs
[ OK ] TestMath/4.abs (0 ms)
[ RUN ] TestMath/4.square
[ OK ] TestMath/4.square (0 ms)
[ RUN ] TestMath/4.sqrtAndInvSqrt
[ OK ] TestMath/4.sqrtAndInvSqrt (0 ms)
[ RUN ] TestMath/4.trig
[ OK ] TestMath/4.trig (0 ms)
[ RUN ] TestMath/4.exponential
[ OK ] TestMath/4.exponential (0 ms)
[----------] 8 tests from TestMath/4 (0 ms total)
[----------] 8 tests from TestMath/5, where TypeParam = std::pair<int, RAJA::policy::cuda::cuda_exec_explicit<32ul, 1ul, false> >
[ RUN ] TestMath/5.numValues
[ OK ] TestMath/5.numValues (0 ms)
[ RUN ] TestMath/5.convert
[ OK ] TestMath/5.convert (164 ms)
[ RUN ] TestMath/5.maxAndMin
[ OK ] TestMath/5.maxAndMin (1 ms)
[ RUN ] TestMath/5.abs
[ OK ] TestMath/5.abs (0 ms)
[ RUN ] TestMath/5.square
[ OK ] TestMath/5.square (0 ms)
[ RUN ] TestMath/5.sqrtAndInvSqrt
[ OK ] TestMath/5.sqrtAndInvSqrt (0 ms)
[ RUN ] TestMath/5.trig
[ OK ] TestMath/5.trig (0 ms)
[ RUN ] TestMath/5.exponential
[ OK ] TestMath/5.exponential (0 ms)
[----------] 8 tests from TestMath/5 (165 ms total)
[----------] 8 tests from TestMath/6, where TypeParam = std::pair<long, RAJA::policy::cuda::cuda_exec_explicit<32ul, 1ul, false> >
[ RUN ] TestMath/6.numValues
[ OK ] TestMath/6.numValues (0 ms)
[ RUN ] TestMath/6.convert
[ OK ] TestMath/6.convert (0 ms)
[ RUN ] TestMath/6.maxAndMin
[ OK ] TestMath/6.maxAndMin (0 ms)
[ RUN ] TestMath/6.abs
[ OK ] TestMath/6.abs (0 ms)
[ RUN ] TestMath/6.square
[ OK ] TestMath/6.square (0 ms)
[ RUN ] TestMath/6.sqrtAndInvSqrt
[ OK ] TestMath/6.sqrtAndInvSqrt (0 ms)
[ RUN ] TestMath/6.trig
[ OK ] TestMath/6.trig (0 ms)
[ RUN ] TestMath/6.exponential
[ OK ] TestMath/6.exponential (0 ms)
[----------] 8 tests from TestMath/6 (0 ms total)
[----------] 8 tests from TestMath/7, where TypeParam = std::pair<long long, RAJA::policy::cuda::cuda_exec_explicit<32ul, 1ul, false> >
[ RUN ] TestMath/7.numValues
[ OK ] TestMath/7.numValues (0 ms)
[ RUN ] TestMath/7.convert
[ OK ] TestMath/7.convert (0 ms)
[ RUN ] TestMath/7.maxAndMin
[ OK ] TestMath/7.maxAndMin (0 ms)
[ RUN ] TestMath/7.abs
[ OK ] TestMath/7.abs (0 ms)
[ RUN ] TestMath/7.square
[ OK ] TestMath/7.square (0 ms)
[ RUN ] TestMath/7.sqrtAndInvSqrt
[ OK ] TestMath/7.sqrtAndInvSqrt (0 ms)
[ RUN ] TestMath/7.trig
[ OK ] TestMath/7.trig (0 ms)
[ RUN ] TestMath/7.exponential
[ OK ] TestMath/7.exponential (0 ms)
[----------] 8 tests from TestMath/7 (0 ms total)
[----------] 8 tests from TestMath/8, where TypeParam = std::pair<float, RAJA::policy::cuda::cuda_exec_explicit<32ul, 1ul, false> >
[ RUN ] TestMath/8.numValues
[ OK ] TestMath/8.numValues (0 ms)
[ RUN ] TestMath/8.convert
[ OK ] TestMath/8.convert (0 ms)
[ RUN ] TestMath/8.maxAndMin
[ OK ] TestMath/8.maxAndMin (0 ms)
[ RUN ] TestMath/8.abs
[ OK ] TestMath/8.abs (0 ms)
[ RUN ] TestMath/8.square
[ OK ] TestMath/8.square (0 ms)
[ RUN ] TestMath/8.sqrtAndInvSqrt
[ OK ] TestMath/8.sqrtAndInvSqrt (0 ms)
[ RUN ] TestMath/8.trig
[ OK ] TestMath/8.trig (0 ms)
[ RUN ] TestMath/8.exponential
[ OK ] TestMath/8.exponential (0 ms)
[----------] 8 tests from TestMath/8 (0 ms total)
[----------] 8 tests from TestMath/9, where TypeParam = std::pair<double, RAJA::policy::cuda::cuda_exec_explicit<32ul, 1ul, false> >
[ RUN ] TestMath/9.numValues
[ OK ] TestMath/9.numValues (0 ms)
[ RUN ] TestMath/9.convert
[ OK ] TestMath/9.convert (0 ms)
[ RUN ] TestMath/9.maxAndMin
[ OK ] TestMath/9.maxAndMin (0 ms)
[ RUN ] TestMath/9.abs
[ OK ] TestMath/9.abs (0 ms)
[ RUN ] TestMath/9.square
[ OK ] TestMath/9.square (0 ms)
[ RUN ] TestMath/9.sqrtAndInvSqrt
[ OK ] TestMath/9.sqrtAndInvSqrt (0 ms)
[ RUN ] TestMath/9.trig
[ OK ] TestMath/9.trig (0 ms)
[ RUN ] TestMath/9.exponential
[ OK ] TestMath/9.exponential (0 ms)
[----------] 8 tests from TestMath/9 (0 ms total)
[----------] 8 tests from TestMath/10, where TypeParam = std::pair<__half, RAJA::policy::cuda::cuda_exec_explicit<32ul, 1ul, false> >
[ RUN ] TestMath/10.numValues
[ OK ] TestMath/10.numValues (0 ms)
[ RUN ] TestMath/10.convert
[ OK ] TestMath/10.convert (0 ms)
[ RUN ] TestMath/10.maxAndMin
[ OK ] TestMath/10.maxAndMin (0 ms)
[ RUN ] TestMath/10.abs
[ OK ] TestMath/10.abs (0 ms)
[ RUN ] TestMath/10.square
[ OK ] TestMath/10.square (0 ms)
[ RUN ] TestMath/10.sqrtAndInvSqrt
[ OK ] TestMath/10.sqrtAndInvSqrt (0 ms)
[ RUN ] TestMath/10.trig
[ OK ] TestMath/10.trig (0 ms)
[ RUN ] TestMath/10.exponential
[ OK ] TestMath/10.exponential (0 ms)
[----------] 8 tests from TestMath/10 (0 ms total)
[----------] 8 tests from TestMath2/0, where TypeParam = std::pair<__half2, RAJA::policy::cuda::cuda_exec_explicit<32ul, 1ul, false> >
[ RUN ] TestMath2/0.numValues
[ OK ] TestMath2/0.numValues (0 ms)
[ RUN ] TestMath2/0.convert
[ OK ] TestMath2/0.convert (0 ms)
[ RUN ] TestMath2/0.maxAndMin
[ OK ] TestMath2/0.maxAndMin (0 ms)
[ RUN ] TestMath2/0.abs
[ OK ] TestMath2/0.abs (0 ms)
[ RUN ] TestMath2/0.square
[ OK ] TestMath2/0.square (0 ms)
[ RUN ] TestMath2/0.sqrtAndInvSqrt
[ OK ] TestMath2/0.sqrtAndInvSqrt (0 ms)
[ RUN ] TestMath2/0.trig
[ OK ] TestMath2/0.trig (0 ms)
[ RUN ] TestMath2/0.exponential
[ OK ] TestMath2/0.exponential (0 ms)
[----------] 8 tests from TestMath2/0 (0 ms total)
[----------] 6 tests from TestHalfMath
[ RUN ] TestHalfMath.asinHalfAccuracy
Max difference is 1.11094322e-03
Max relative difference is 5.73841195e-03
[ OK ] TestHalfMath.asinHalfAccuracy (14 ms)
[ RUN ] TestHalfMath.asinHalf2Accuracy
Max difference is 1.11094322e-03
Max relative difference is 5.73841195e-03
[ OK ] TestHalfMath.asinHalf2Accuracy (15 ms)
[ RUN ] TestHalfMath.acosHalfAccuracy
Max difference is 2.64615570e-03
Max relative difference is 1.45285107e-03
[ OK ] TestHalfMath.acosHalfAccuracy (10 ms)
[ RUN ] TestHalfMath.acosHalf2Accuracy
Max difference is 2.64615570e-03
Max relative difference is 1.45285107e-03
[ OK ] TestHalfMath.acosHalf2Accuracy (15 ms)
[ RUN ] TestHalfMath.atan2HalfAccuracy
/appli_RD/LACOSTE/GEOSX/GEOSX/src/coreComponents/LvArray/unitTests/testMath.cpp:569: Failure
Expected: (maxDiff.get()) < (3 * float( NumericLimitsNC< __half >{}.epsilon )), actual: 6.28222 vs 0.00292969
Max difference is 6.28221765e+00
Max relative difference is 1.24008826e-03
[ FAILED ] TestHalfMath.atan2HalfAccuracy (47 ms)
[ RUN ] TestHalfMath.atan2Half2Accuracy
/appli_RD/LACOSTE/GEOSX/GEOSX/src/coreComponents/LvArray/unitTests/testMath.cpp:627: Failure
Expected: (maxDiff.get()) < (3 * float( NumericLimitsNC< __half >{}.epsilon )), actual: 6.28222 vs 0.00292969
Max difference is 6.28221774e+00
Max relative difference is 1.50966642e-03
[ FAILED ] TestHalfMath.atan2Half2Accuracy (39 ms)
[----------] 6 tests from TestHalfMath (140 ms total)
[----------] Global test environment tear-down
[==========] 102 tests from 13 test suites ran. (496 ms total)
[ PASSED ] 100 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] TestHalfMath.atan2HalfAccuracy
[ FAILED ] TestHalfMath.atan2Half2Accuracy
2 FAILED TESTS
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[34039,1],0]
Exit code: 1
--------------------------------------------------------------------------
Start 107: testDruckerPrager
2/11 Test GEOS-DEV/thirdPartyLibs#107: testDruckerPrager ...........................***Failed 3.41 sec
Running main() from /appli_RD/LACOSTE/GEOSX/GEOSX/src/cmake/blt/thirdparty_builtin/googletest-master-2020-01-07/googletest/src/gtest_main.cc
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from DruckerPragerTests
[ RUN ] DruckerPragerTests.testDruckerPragerHost
Moved 0.4 KB to the DEVICE: LvArray::Array<double, 3, camp::int_seq<long, 2l, 1l, 0l>, int, LvArray::ChaiBuffer> root/discretization/granite_stress
Moved 0.4 KB to the DEVICE: LvArray::Array<double, 3, camp::int_seq<long, 2l, 1l, 0l>, int, LvArray::ChaiBuffer> root/discretization/granite_oldStress
Moved 0.0 KB to the DEVICE: LvArray::Array<double, 1, camp::int_seq<long, 0l>, int, LvArray::ChaiBuffer> root/discretization/granite_bulkModulus
Moved 0.0 KB to the DEVICE: LvArray::Array<double, 1, camp::int_seq<long, 0l>, int, LvArray::ChaiBuffer> root/discretization/granite_shearModulus
Moved 0.0 KB to the DEVICE: LvArray::Array<double, 1, camp::int_seq<long, 0l>, int, LvArray::ChaiBuffer> root/discretization/granite_friction
Moved 0.0 KB to the DEVICE: LvArray::Array<double, 1, camp::int_seq<long, 0l>, int, LvArray::ChaiBuffer> root/discretization/granite_dilation
Moved 0.0 KB to the DEVICE: LvArray::Array<double, 1, camp::int_seq<long, 0l>, int, LvArray::ChaiBuffer> root/discretization/granite_hardening
Moved 0.1 KB to the DEVICE: LvArray::Array<double, 2, camp::int_seq<long, 0l, 1l>, int, LvArray::ChaiBuffer> root/discretization/granite_cohesion
Moved 0.1 KB to the DEVICE: LvArray::Array<double, 2, camp::int_seq<long, 0l, 1l>, int, LvArray::ChaiBuffer> root/discretization/granite_oldCohesion
terminate called without an active exception
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node p3dev01 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
Start 109: testModifiedCamClay
3/11 Test GEOS-DEV/thirdPartyLibs#109: testModifiedCamClay .........................***Failed 3.26 sec
Running main() from /appli_RD/LACOSTE/GEOSX/GEOSX/src/cmake/blt/thirdparty_builtin/googletest-master-2020-01-07/googletest/src/gtest_main.cc
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from ModifiedCamClayTests
[ RUN ] ModifiedCamClayTests.testModifiedCamClayHost
Moved 0.4 KB to the DEVICE: LvArray::Array<double, 3, camp::int_seq<long, 2l, 1l, 0l>, int, LvArray::ChaiBuffer> root/discretization/granite_stress
Moved 0.4 KB to the DEVICE: LvArray::Array<double, 3, camp::int_seq<long, 2l, 1l, 0l>, int, LvArray::ChaiBuffer> root/discretization/granite_oldStress
Moved 0.0 KB to the DEVICE: LvArray::Array<double, 1, camp::int_seq<long, 0l>, int, LvArray::ChaiBuffer> root/discretization/granite_recompressionIndex
Moved 0.0 KB to the DEVICE: LvArray::Array<double, 1, camp::int_seq<long, 0l>, int, LvArray::ChaiBuffer> root/discretization/granite_shearModulus
Moved 0.0 KB to the DEVICE: LvArray::Array<double, 1, camp::int_seq<long, 0l>, int, LvArray::ChaiBuffer> root/discretization/granite_virginCompressionIndex
Moved 0.0 KB to the DEVICE: LvArray::Array<double, 1, camp::int_seq<long, 0l>, int, LvArray::ChaiBuffer> root/discretization/granite_cslSlope
Moved 0.1 KB to the DEVICE: LvArray::Array<double, 2, camp::int_seq<long, 0l, 1l>, int, LvArray::ChaiBuffer> root/discretization/granite_preConsolidationPressure
Moved 0.1 KB to the DEVICE: LvArray::Array<double, 2, camp::int_seq<long, 0l, 1l>, int, LvArray::ChaiBuffer> root/discretization/granite_oldPreConsolidationPressure
terminate called without an active exception
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node p3dev01 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
Start 149: testTriaxial_druckerPragerExtended
4/11 Test GEOS-DEV/thirdPartyLibs#149: testTriaxial_druckerPragerExtended ..........***Failed 12.64 sec
Max threads: 4
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from testTriaxial
[ RUN ] testTriaxial.testTriaxial
Adding Event: SoloEvent, triaxialDriver
TableFunction: strainFunction
TableFunction: stressFunction
Adding Mesh: InternalMesh, mesh1
Adding Object CellElementRegion named dummy from ObjectManager::Catalog.
Total number of nodes:8
Total number of elems:1
Rank 0: Total number of nodes:8
dummy/cellBlock01 does not have a discretization associated with it.
Time: 0s, dt:1s, Cycle: 0
unknown file: Failure
C++ exception with description "campCudaErrchk(cudaStreamSynchronize(stream)) an illegal memory access was encountered /workrd/SCR/NUM/xavier/p3/tpl2/install-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/raja/include/camp/resource/cuda.hpp:172" thrown in the test body.
[ FAILED ] testTriaxial.testTriaxial (2504 ms)
[----------] 1 test from testTriaxial (2504 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (2505 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] testTriaxial.testTriaxial
1 FAILED TEST
Umpire DEVICE sum across ranks: 3.0 KB
Umpire DEVICE rank max: 3.0 KB
Umpire DEVICE::0 sum across ranks: 3.0 KB
Umpire DEVICE::0 rank max: 3.0 KB
Umpire HOST sum across ranks: 11.7 KB
Umpire HOST rank max: 11.7 KB
terminate called after throwing an instance of 'umpire::util::Exception'
what(): ! Umpire Exception [/work206/workrd/SCR/NUM/xavier/p3/tpl2/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/chai/src/chai/src/tpl/umpire/src/umpire/alloc/CudaMallocAllocator.hpp:54]: deallocate cudaFree( ptr = 0x20009d600400 ) failed with error: an illegal memory access was encountered
Backtrace: 21 frames
0 0x132c0a3c No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial() [0x132c0a3c]
1 0x132c194c No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN6umpire5alloc19CudaMallocAllocator10deallocateEPv+0x3bc) [0x132c194c]
2 0x132c0e3c No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN6umpire8resource24CudaDeviceMemoryResource10deallocateEPvm+0x36c) [0x132c0e3c]
3 0x114de930 No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN6umpire9Allocator10deallocateEPv+0x370) [0x114de930]
4 0x13254594 No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN4chai12ArrayManager4freeEPNS_13PointerRecordENS_14ExecutionSpaceE+0x564) [0x13254594]
5 0x10df481c No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN7LvArray10ChaiBufferIdE4freeEv+0x8c) [0x10df481c]
6 0x110a8c10 No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive21DruckerPragerExtendedD1Ev+0x30) [0x110a8c10]
7 0x110a8cb0 No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive21DruckerPragerExtendedD0Ev+0x20) [0x110a8cb0]
8 0x10dcf57c No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c) [0x10dcf57c]
9 0x10e7bf84 No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD1Ev+0x24) [0x10e7bf84]
10 0x10e7bfd0 No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD0Ev+0x20) [0x10e7bfd0]
11 0x10dcf57c No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c) [0x10dcf57c]
12 0x114d8cfc No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD1Ev+0x1bc) [0x114d8cfc]
13 0x114d8d80 No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD0Ev+0x20) [0x114d8d80]
14 0x10dcf57c No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c) [0x10dcf57c]
15 0x12e37aec No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD1Ev+0x2dc) [0x12e37aec]
16 0x12e37b40 No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD0Ev+0x20) [0x12e37b40]
17 0x12e3473c No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx10GeosxStateD1Ev+0x42c) [0x12e3473c]
18 0x10c7391c No dladdr: /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(main+0xac) [0x10c7391c]
19 0x2000094841e8 No dladdr: /lib64/power9/libc.so.6(+0x241e8) [0x2000094841e8]
20 0x2000094843d4 No dladdr: /lib64/power9/libc.so.6(__libc_start_main+0xb4) [0x2000094843d4]
[p3dev01:32308] *** Process received signal ***
[p3dev01:32308] Signal: Aborted (6)
[p3dev01:32308] Associated errno: Unknown error 8192 (8192)
[p3dev01:32308] Signal code: (154644072)
[p3dev01:32308] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8]
[p3dev01:32308] [ 1] /lib64/power9/libc.so.6(gsignal+0xd8)[0x2000094a3708]
[p3dev01:32308] [ 2] /lib64/power9/libc.so.6(abort+0x164)[0x200009483bcc]
[p3dev01:32308] [ 3] /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x158)[0x200009234a98]
[p3dev01:32308] [ 4] /lib64/libstdc++.so.6(+0xaf6d4)[0x20000922f6d4]
[p3dev01:32308] [ 5] /lib64/libstdc++.so.6(+0xadb18)[0x20000922db18]
[p3dev01:32308] [ 6] /lib64/libstdc++.so.6(__gxx_personality_v0+0x40c)[0x20000922ebcc]
[p3dev01:32308] [ 7] /lib64/libgcc_s.so.1(+0xc210)[0x20000942c210]
[p3dev01:32308] [ 8] /lib64/libgcc_s.so.1(_Unwind_Resume+0x178)[0x20000942ce38]
[p3dev01:32308] [ 9] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN7LvArray10ChaiBufferIdE4freeEv+0x158)[0x10df48e8]
[p3dev01:32308] [10] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive21DruckerPragerExtendedD1Ev+0x30)[0x110a8c10]
[p3dev01:32308] [11] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive21DruckerPragerExtendedD0Ev+0x20)[0x110a8cb0]
[p3dev01:32308] [12] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32308] [13] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD1Ev+0x24)[0x10e7bf84]
[p3dev01:32308] [14] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD0Ev+0x20)[0x10e7bfd0]
[p3dev01:32308] [15] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32308] [16] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD1Ev+0x1bc)[0x114d8cfc]
[p3dev01:32308] [17] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD0Ev+0x20)[0x114d8d80]
[p3dev01:32308] [18] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32308] [19] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD1Ev+0x2dc)[0x12e37aec]
[p3dev01:32308] [20] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD0Ev+0x20)[0x12e37b40]
[p3dev01:32308] [21] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx10GeosxStateD1Ev+0x42c)[0x12e3473c]
[p3dev01:32308] [22] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(main+0xac)[0x10c7391c]
[p3dev01:32308] [23] /lib64/power9/libc.so.6(+0x241e8)[0x2000094841e8]
[p3dev01:32308] [24] /lib64/power9/libc.so.6(__libc_start_main+0xb4)[0x2000094843d4]
[p3dev01:32308] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node p3dev01 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
Start 152: testTriaxial_modifiedCamClay
5/11 Test GEOS-DEV/thirdPartyLibs#152: testTriaxial_modifiedCamClay ................***Failed 4.14 sec
Max threads: 4
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from testTriaxial
[ RUN ] testTriaxial.testTriaxial
Adding Event: SoloEvent, triaxialDriver
TableFunction: strainFunction
TableFunction: stressFunction
Adding Mesh: InternalMesh, mesh1
Adding Object CellElementRegion named dummy from ObjectManager::Catalog.
Total number of nodes:8
Total number of elems:1
Rank 0: Total number of nodes:8
dummy/cellBlock01 does not have a discretization associated with it.
Time: 0s, dt:1s, Cycle: 0
unknown file: Failure
C++ exception with description "campCudaErrchk(cudaStreamSynchronize(stream)) an illegal memory access was encountered /workrd/SCR/NUM/xavier/p3/tpl2/install-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/raja/include/camp/resource/cuda.hpp:172" thrown in the test body.
[ FAILED ] testTriaxial.testTriaxial (950 ms)
[----------] 1 test from testTriaxial (950 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (950 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] testTriaxial.testTriaxial
1 FAILED TEST
Umpire DEVICE sum across ranks: 3.7 KB
Umpire DEVICE rank max: 3.7 KB
Umpire DEVICE::0 sum across ranks: 3.7 KB
Umpire DEVICE::0 rank max: 3.7 KB
Umpire HOST sum across ranks: 12.3 KB
Umpire HOST rank max: 12.3 KB
terminate called without an active exception
[p3dev01:32372] *** Process received signal ***
[p3dev01:32372] Signal: Aborted (6)
[p3dev01:32372] Signal code: User function (kill, sigsend, abort, etc.) (0)
[p3dev01:32372] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8]
[p3dev01:32372] [ 1] /lib64/power9/libc.so.6(gsignal+0xd8)[0x2000094a3708]
[p3dev01:32372] [ 2] /lib64/power9/libc.so.6(abort+0x164)[0x200009483bcc]
[p3dev01:32372] [ 3] /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x158)[0x200009234a98]
[p3dev01:32372] [ 4] /lib64/libstdc++.so.6(+0xaf6d4)[0x20000922f6d4]
[p3dev01:32372] [ 5] /lib64/libstdc++.so.6(_ZSt9terminatev+0x20)[0x20000922f790]
[p3dev01:32372] [ 6] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive15ModifiedCamClayD1Ev+0x380)[0x110ac120]
[p3dev01:32372] [ 7] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive15ModifiedCamClayD0Ev+0x20)[0x110ac170]
[p3dev01:32372] [ 8] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32372] [ 9] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD1Ev+0x24)[0x10e7bf84]
[p3dev01:32372] [10] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD0Ev+0x20)[0x10e7bfd0]
[p3dev01:32372] [11] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32372] [12] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD1Ev+0x1bc)[0x114d8cfc]
[p3dev01:32372] [13] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD0Ev+0x20)[0x114d8d80]
[p3dev01:32372] [14] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32372] [15] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD1Ev+0x2dc)[0x12e37aec]
[p3dev01:32372] [16] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD0Ev+0x20)[0x12e37b40]
[p3dev01:32372] [17] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx10GeosxStateD1Ev+0x42c)[0x12e3473c]
[p3dev01:32372] [18] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(main+0xac)[0x10c7391c]
[p3dev01:32372] [19] /lib64/power9/libc.so.6(+0x241e8)[0x2000094841e8]
[p3dev01:32372] [20] /lib64/power9/libc.so.6(__libc_start_main+0xb4)[0x2000094843d4]
[p3dev01:32372] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node p3dev01 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
Start 153: testTriaxial_delftEggUseLinear
6/11 Test GEOS-DEV/thirdPartyLibs#153: testTriaxial_delftEggUseLinear ..............***Failed 3.95 sec
Max threads: 4
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from testTriaxial
[ RUN ] testTriaxial.testTriaxial
Adding Event: SoloEvent, triaxialDriver
TableFunction: strainFunction
TableFunction: stressFunction
Adding Mesh: InternalMesh, mesh1
Adding Object CellElementRegion named dummy from ObjectManager::Catalog.
Total number of nodes:8
Total number of elems:1
Rank 0: Total number of nodes:8
dummy/cellBlock01 does not have a discretization associated with it.
Time: 0s, dt:1s, Cycle: 0
unknown file: Failure
C++ exception with description "campCudaErrchk(cudaStreamSynchronize(stream)) an illegal memory access was encountered /workrd/SCR/NUM/xavier/p3/tpl2/install-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/raja/include/camp/resource/cuda.hpp:172" thrown in the test body.
[ FAILED ] testTriaxial.testTriaxial (859 ms)
[----------] 1 test from testTriaxial (859 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (859 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] testTriaxial.testTriaxial
1 FAILED TEST
Umpire DEVICE sum across ranks: 3.7 KB
Umpire DEVICE rank max: 3.7 KB
Umpire DEVICE::0 sum across ranks: 3.7 KB
Umpire DEVICE::0 rank max: 3.7 KB
Umpire HOST sum across ranks: 12.3 KB
Umpire HOST rank max: 12.3 KB
terminate called without an active exception
[p3dev01:32416] *** Process received signal ***
[p3dev01:32416] Signal: Aborted (6)
[p3dev01:32416] Signal code: User function (kill, sigsend, abort, etc.) (0)
[p3dev01:32416] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8]
[p3dev01:32416] [ 1] /lib64/power9/libc.so.6(gsignal+0xd8)[0x2000094a3708]
[p3dev01:32416] [ 2] /lib64/power9/libc.so.6(abort+0x164)[0x200009483bcc]
[p3dev01:32416] [ 3] /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x158)[0x200009234a98]
[p3dev01:32416] [ 4] /lib64/libstdc++.so.6(+0xaf6d4)[0x20000922f6d4]
[p3dev01:32416] [ 5] /lib64/libstdc++.so.6(_ZSt9terminatev+0x20)[0x20000922f790]
[p3dev01:32416] [ 6] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive8DelftEggD1Ev+0x4e0)[0x110af9e0]
[p3dev01:32416] [ 7] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive8DelftEggD0Ev+0x20)[0x110afa40]
[p3dev01:32416] [ 8] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32416] [ 9] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD1Ev+0x24)[0x10e7bf84]
[p3dev01:32416] [10] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD0Ev+0x20)[0x10e7bfd0]
[p3dev01:32416] [11] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32416] [12] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD1Ev+0x1bc)[0x114d8cfc]
[p3dev01:32416] [13] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD0Ev+0x20)[0x114d8d80]
[p3dev01:32416] [14] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32416] [15] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD1Ev+0x2dc)[0x12e37aec]
[p3dev01:32416] [16] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD0Ev+0x20)[0x12e37b40]
[p3dev01:32416] [17] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx10GeosxStateD1Ev+0x42c)[0x12e3473c]
[p3dev01:32416] [18] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(main+0xac)[0x10c7391c]
[p3dev01:32416] [19] /lib64/power9/libc.so.6(+0x241e8)[0x2000094841e8]
[p3dev01:32416] [20] /lib64/power9/libc.so.6(__libc_start_main+0xb4)[0x2000094843d4]
[p3dev01:32416] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node p3dev01 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
Start 154: testTriaxial_modifiedCamClayVolumetric
7/11 Test GEOS-DEV/thirdPartyLibs#154: testTriaxial_modifiedCamClayVolumetric ......***Failed 4.00 sec
Max threads: 4
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from testTriaxial
[ RUN ] testTriaxial.testTriaxial
Adding Event: SoloEvent, triaxialDriver
TableFunction: strainFunction
Adding Mesh: InternalMesh, mesh1
Adding Object CellElementRegion named dummy from ObjectManager::Catalog.
Total number of nodes:8
Total number of elems:1
Rank 0: Total number of nodes:8
dummy/cellBlock01 does not have a discretization associated with it.
Time: 0s, dt:1s, Cycle: 0
unknown file: Failure
C++ exception with description "campCudaErrchk(cudaStreamSynchronize(stream)) an illegal memory access was encountered /workrd/SCR/NUM/xavier/p3/tpl2/install-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/raja/include/camp/resource/cuda.hpp:172" thrown in the test body.
[ FAILED ] testTriaxial.testTriaxial (902 ms)
[----------] 1 test from testTriaxial (902 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (902 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] testTriaxial.testTriaxial
1 FAILED TEST
Umpire DEVICE sum across ranks: 3.7 KB
Umpire DEVICE rank max: 3.7 KB
Umpire DEVICE::0 sum across ranks: 3.7 KB
Umpire DEVICE::0 rank max: 3.7 KB
Umpire HOST sum across ranks: 12.1 KB
Umpire HOST rank max: 12.1 KB
terminate called without an active exception
[p3dev01:32457] *** Process received signal ***
[p3dev01:32457] Signal: Aborted (6)
[p3dev01:32457] Signal code: User function (kill, sigsend, abort, etc.) (0)
[p3dev01:32457] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8]
[p3dev01:32457] [ 1] /lib64/power9/libc.so.6(gsignal+0xd8)[0x2000094a3708]
[p3dev01:32457] [ 2] /lib64/power9/libc.so.6(abort+0x164)[0x200009483bcc]
[p3dev01:32457] [ 3] /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x158)[0x200009234a98]
[p3dev01:32457] [ 4] /lib64/libstdc++.so.6(+0xaf6d4)[0x20000922f6d4]
[p3dev01:32457] [ 5] /lib64/libstdc++.so.6(_ZSt9terminatev+0x20)[0x20000922f790]
[p3dev01:32457] [ 6] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive15ModifiedCamClayD1Ev+0x380)[0x110ac120]
[p3dev01:32457] [ 7] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive15ModifiedCamClayD0Ev+0x20)[0x110ac170]
[p3dev01:32457] [ 8] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32457] [ 9] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD1Ev+0x24)[0x10e7bf84]
[p3dev01:32457] [10] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD0Ev+0x20)[0x10e7bfd0]
[p3dev01:32457] [11] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32457] [12] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD1Ev+0x1bc)[0x114d8cfc]
[p3dev01:32457] [13] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD0Ev+0x20)[0x114d8d80]
[p3dev01:32457] [14] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32457] [15] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD1Ev+0x2dc)[0x12e37aec]
[p3dev01:32457] [16] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD0Ev+0x20)[0x12e37b40]
[p3dev01:32457] [17] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx10GeosxStateD1Ev+0x42c)[0x12e3473c]
[p3dev01:32457] [18] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(main+0xac)[0x10c7391c]
[p3dev01:32457] [19] /lib64/power9/libc.so.6(+0x241e8)[0x2000094841e8]
[p3dev01:32457] [20] /lib64/power9/libc.so.6(__libc_start_main+0xb4)[0x2000094843d4]
[p3dev01:32457] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node p3dev01 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
Start 155: testTriaxial_delftEggLoadPathDryUseLinear
8/11 Test GEOS-DEV/thirdPartyLibs#155: testTriaxial_delftEggLoadPathDryUseLinear ...***Failed 4.01 sec
Max threads: 4
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from testTriaxial
[ RUN ] testTriaxial.testTriaxial
Adding Event: SoloEvent, triaxialDriver
TableFunction: strainFunction
TableFunction: stressFunction
Adding Mesh: InternalMesh, mesh1
Adding Object CellElementRegion named dummy from ObjectManager::Catalog.
Total number of nodes:8
Total number of elems:1
Rank 0: Total number of nodes:8
dummy/cellBlock01 does not have a discretization associated with it.
Time: 0s, dt:1s, Cycle: 0
unknown file: Failure
C++ exception with description "campCudaErrchk(cudaStreamSynchronize(stream)) an illegal memory access was encountered /workrd/SCR/NUM/xavier/p3/tpl2/install-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/raja/include/camp/resource/cuda.hpp:172" thrown in the test body.
[ FAILED ] testTriaxial.testTriaxial (844 ms)
[----------] 1 test from testTriaxial (844 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (844 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] testTriaxial.testTriaxial
1 FAILED TEST
Umpire DEVICE sum across ranks: 3.7 KB
Umpire DEVICE rank max: 3.7 KB
Umpire DEVICE::0 sum across ranks: 3.7 KB
Umpire DEVICE::0 rank max: 3.7 KB
Umpire HOST sum across ranks: 12.3 KB
Umpire HOST rank max: 12.3 KB
terminate called without an active exception
[p3dev01:32504] *** Process received signal ***
[p3dev01:32504] Signal: Aborted (6)
[p3dev01:32504] Signal code: User function (kill, sigsend, abort, etc.) (0)
[p3dev01:32504] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8]
[p3dev01:32504] [ 1] /lib64/power9/libc.so.6(gsignal+0xd8)[0x2000094a3708]
[p3dev01:32504] [ 2] /lib64/power9/libc.so.6(abort+0x164)[0x200009483bcc]
[p3dev01:32504] [ 3] /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x158)[0x200009234a98]
[p3dev01:32504] [ 4] /lib64/libstdc++.so.6(+0xaf6d4)[0x20000922f6d4]
[p3dev01:32504] [ 5] /lib64/libstdc++.so.6(_ZSt9terminatev+0x20)[0x20000922f790]
[p3dev01:32504] [ 6] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive8DelftEggD1Ev+0x4e0)[0x110af9e0]
[p3dev01:32504] [ 7] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive8DelftEggD0Ev+0x20)[0x110afa40]
[p3dev01:32504] [ 8] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32504] [ 9] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD1Ev+0x24)[0x10e7bf84]
[p3dev01:32504] [10] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD0Ev+0x20)[0x10e7bfd0]
[p3dev01:32504] [11] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32504] [12] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD1Ev+0x1bc)[0x114d8cfc]
[p3dev01:32504] [13] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD0Ev+0x20)[0x114d8d80]
[p3dev01:32504] [14] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32504] [15] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD1Ev+0x2dc)[0x12e37aec]
[p3dev01:32504] [16] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD0Ev+0x20)[0x12e37b40]
[p3dev01:32504] [17] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx10GeosxStateD1Ev+0x42c)[0x12e3473c]
[p3dev01:32504] [18] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(main+0xac)[0x10c7391c]
[p3dev01:32504] [19] /lib64/power9/libc.so.6(+0x241e8)[0x2000094841e8]
[p3dev01:32504] [20] /lib64/power9/libc.so.6(__libc_start_main+0xb4)[0x2000094843d4]
[p3dev01:32504] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node p3dev01 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
Start 156: testTriaxial_delftEggLoadPathWetUseLinear
9/11 Test GEOS-DEV/thirdPartyLibs#156: testTriaxial_delftEggLoadPathWetUseLinear ...***Failed 3.99 sec
Max threads: 4
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from testTriaxial
[ RUN ] testTriaxial.testTriaxial
Adding Event: SoloEvent, triaxialDriver
TableFunction: strainFunction
TableFunction: stressFunction
Adding Mesh: InternalMesh, mesh1
Adding Object CellElementRegion named dummy from ObjectManager::Catalog.
Total number of nodes:8
Total number of elems:1
Rank 0: Total number of nodes:8
dummy/cellBlock01 does not have a discretization associated with it.
Time: 0s, dt:1s, Cycle: 0
unknown file: Failure
C++ exception with description "campCudaErrchk(cudaStreamSynchronize(stream)) an illegal memory access was encountered /workrd/SCR/NUM/xavier/p3/tpl2/install-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/raja/include/camp/resource/cuda.hpp:172" thrown in the test body.
[ FAILED ] testTriaxial.testTriaxial (822 ms)
[----------] 1 test from testTriaxial (822 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (822 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] testTriaxial.testTriaxial
1 FAILED TEST
Umpire DEVICE sum across ranks: 3.7 KB
Umpire DEVICE rank max: 3.7 KB
Umpire DEVICE::0 sum across ranks: 3.7 KB
Umpire DEVICE::0 rank max: 3.7 KB
Umpire HOST sum across ranks: 12.3 KB
Umpire HOST rank max: 12.3 KB
terminate called without an active exception
[p3dev01:32550] *** Process received signal ***
[p3dev01:32550] Signal: Aborted (6)
[p3dev01:32550] Signal code: User function (kill, sigsend, abort, etc.) (0)
[p3dev01:32550] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8]
[p3dev01:32550] [ 1] /lib64/power9/libc.so.6(gsignal+0xd8)[0x2000094a3708]
[p3dev01:32550] [ 2] /lib64/power9/libc.so.6(abort+0x164)[0x200009483bcc]
[p3dev01:32550] [ 3] /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x158)[0x200009234a98]
[p3dev01:32550] [ 4] /lib64/libstdc++.so.6(+0xaf6d4)[0x20000922f6d4]
[p3dev01:32550] [ 5] /lib64/libstdc++.so.6(_ZSt9terminatev+0x20)[0x20000922f790]
[p3dev01:32550] [ 6] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive8DelftEggD1Ev+0x4e0)[0x110af9e0]
[p3dev01:32550] [ 7] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive8DelftEggD0Ev+0x20)[0x110afa40]
[p3dev01:32550] [ 8] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32550] [ 9] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD1Ev+0x24)[0x10e7bf84]
[p3dev01:32550] [10] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD0Ev+0x20)[0x10e7bfd0]
[p3dev01:32550] [11] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32550] [12] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD1Ev+0x1bc)[0x114d8cfc]
[p3dev01:32550] [13] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD0Ev+0x20)[0x114d8d80]
[p3dev01:32550] [14] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32550] [15] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD1Ev+0x2dc)[0x12e37aec]
[p3dev01:32550] [16] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD0Ev+0x20)[0x12e37b40]
[p3dev01:32550] [17] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx10GeosxStateD1Ev+0x42c)[0x12e3473c]
[p3dev01:32550] [18] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(main+0xac)[0x10c7391c]
[p3dev01:32550] [19] /lib64/power9/libc.so.6(+0x241e8)[0x2000094841e8]
[p3dev01:32550] [20] /lib64/power9/libc.so.6(__libc_start_main+0xb4)[0x2000094843d4]
[p3dev01:32550] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node p3dev01 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
Start 157: testTriaxial_delftEggCase1
10/11 Test GEOS-DEV/thirdPartyLibs#157: testTriaxial_delftEggCase1 ..................***Failed 4.06 sec
Max threads: 4
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from testTriaxial
[ RUN ] testTriaxial.testTriaxial
Adding Event: SoloEvent, triaxialDriver
TableFunction: strainFunction
TableFunction: stressFunction
Adding Mesh: InternalMesh, mesh1
Adding Object CellElementRegion named dummy from ObjectManager::Catalog.
Total number of nodes:8
Total number of elems:1
Rank 0: Total number of nodes:8
dummy/cellBlock01 does not have a discretization associated with it.
Time: 0s, dt:1s, Cycle: 0
unknown file: Failure
C++ exception with description "campCudaErrchk(cudaStreamSynchronize(stream)) an illegal memory access was encountered /workrd/SCR/NUM/xavier/p3/tpl2/install-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/raja/include/camp/resource/cuda.hpp:172" thrown in the test body.
[ FAILED ] testTriaxial.testTriaxial (935 ms)
[----------] 1 test from testTriaxial (935 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (935 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] testTriaxial.testTriaxial
1 FAILED TEST
Umpire DEVICE sum across ranks: 3.7 KB
Umpire DEVICE rank max: 3.7 KB
Umpire DEVICE::0 sum across ranks: 3.7 KB
Umpire DEVICE::0 rank max: 3.7 KB
Umpire HOST sum across ranks: 12.3 KB
Umpire HOST rank max: 12.3 KB
terminate called without an active exception
[p3dev01:32590] *** Process received signal ***
[p3dev01:32590] Signal: Aborted (6)
[p3dev01:32590] Signal code: User function (kill, sigsend, abort, etc.) (0)
[p3dev01:32590] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8]
[p3dev01:32590] [ 1] /lib64/power9/libc.so.6(gsignal+0xd8)[0x2000094a3708]
[p3dev01:32590] [ 2] /lib64/power9/libc.so.6(abort+0x164)[0x200009483bcc]
[p3dev01:32590] [ 3] /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x158)[0x200009234a98]
[p3dev01:32590] [ 4] /lib64/libstdc++.so.6(+0xaf6d4)[0x20000922f6d4]
[p3dev01:32590] [ 5] /lib64/libstdc++.so.6(_ZSt9terminatev+0x20)[0x20000922f790]
[p3dev01:32590] [ 6] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive8DelftEggD1Ev+0x4e0)[0x110af9e0]
[p3dev01:32590] [ 7] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive8DelftEggD0Ev+0x20)[0x110afa40]
[p3dev01:32590] [ 8] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32590] [ 9] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD1Ev+0x24)[0x10e7bf84]
[p3dev01:32590] [10] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD0Ev+0x20)[0x10e7bfd0]
[p3dev01:32590] [11] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32590] [12] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD1Ev+0x1bc)[0x114d8cfc]
[p3dev01:32590] [13] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD0Ev+0x20)[0x114d8d80]
[p3dev01:32590] [14] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32590] [15] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD1Ev+0x2dc)[0x12e37aec]
[p3dev01:32590] [16] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD0Ev+0x20)[0x12e37b40]
[p3dev01:32590] [17] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx10GeosxStateD1Ev+0x42c)[0x12e3473c]
[p3dev01:32590] [18] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(main+0xac)[0x10c7391c]
[p3dev01:32590] [19] /lib64/power9/libc.so.6(+0x241e8)[0x2000094841e8]
[p3dev01:32590] [20] /lib64/power9/libc.so.6(__libc_start_main+0xb4)[0x2000094843d4]
[p3dev01:32590] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node p3dev01 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
Start 158: testTriaxial_delftEggCase2
11/11 Test GEOS-DEV/thirdPartyLibs#158: testTriaxial_delftEggCase2 ..................***Failed 4.02 sec
Max threads: 4
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from testTriaxial
[ RUN ] testTriaxial.testTriaxial
Adding Event: SoloEvent, triaxialDriver
TableFunction: strainFunction
TableFunction: stressFunction
Adding Mesh: InternalMesh, mesh1
Adding Object CellElementRegion named dummy from ObjectManager::Catalog.
Total number of nodes:8
Total number of elems:1
Rank 0: Total number of nodes:8
dummy/cellBlock01 does not have a discretization associated with it.
Time: 0s, dt:1s, Cycle: 0
unknown file: Failure
C++ exception with description "campCudaErrchk(cudaStreamSynchronize(stream)) an illegal memory access was encountered /workrd/SCR/NUM/xavier/p3/tpl2/install-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/raja/include/camp/resource/cuda.hpp:172" thrown in the test body.
[ FAILED ] testTriaxial.testTriaxial (917 ms)
[----------] 1 test from testTriaxial (917 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (917 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] testTriaxial.testTriaxial
1 FAILED TEST
Umpire DEVICE sum across ranks: 3.7 KB
Umpire DEVICE rank max: 3.7 KB
Umpire DEVICE::0 sum across ranks: 3.7 KB
Umpire DEVICE::0 rank max: 3.7 KB
Umpire HOST sum across ranks: 12.3 KB
Umpire HOST rank max: 12.3 KB
terminate called without an active exception
[p3dev01:32630] *** Process received signal ***
[p3dev01:32630] Signal: Aborted (6)
[p3dev01:32630] Signal code: User function (kill, sigsend, abort, etc.) (0)
[p3dev01:32630] [ 0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x2000000604d8]
[p3dev01:32630] [ 1] /lib64/power9/libc.so.6(gsignal+0xd8)[0x2000094a3708]
[p3dev01:32630] [ 2] /lib64/power9/libc.so.6(abort+0x164)[0x200009483bcc]
[p3dev01:32630] [ 3] /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x158)[0x200009234a98]
[p3dev01:32630] [ 4] /lib64/libstdc++.so.6(+0xaf6d4)[0x20000922f6d4]
[p3dev01:32630] [ 5] /lib64/libstdc++.so.6(_ZSt9terminatev+0x20)[0x20000922f790]
[p3dev01:32630] [ 6] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive8DelftEggD1Ev+0x4e0)[0x110af9e0]
[p3dev01:32630] [ 7] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive8DelftEggD0Ev+0x20)[0x110afa40]
[p3dev01:32630] [ 8] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32630] [ 9] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD1Ev+0x24)[0x10e7bf84]
[p3dev01:32630] [10] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx12constitutive19ConstitutiveManagerD0Ev+0x20)[0x10e7bfd0]
[p3dev01:32630] [11] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32630] [12] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD1Ev+0x1bc)[0x114d8cfc]
[p3dev01:32630] [13] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx15DomainPartitionD0Ev+0x20)[0x114d8d80]
[p3dev01:32630] [14] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14dataRepository5GroupD1Ev+0x95c)[0x10dcf57c]
[p3dev01:32630] [15] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD1Ev+0x2dc)[0x12e37aec]
[p3dev01:32630] [16] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx14ProblemManagerD0Ev+0x20)[0x12e37b40]
[p3dev01:32630] [17] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(_ZN5geosx10GeosxStateD1Ev+0x42c)[0x12e3473c]
[p3dev01:32630] [18] /work206/workrd/SCR/NUM/xavier/p3/geosx/build-pangea3-gcc8.4.1-openmpi-4.1.2-relwithdebinfo/tests/testTriaxial(main+0xac)[0x10c7391c]
[p3dev01:32630] [19] /lib64/power9/libc.so.6(+0x241e8)[0x2000094841e8]
[p3dev01:32630] [20] /lib64/power9/libc.so.6(__libc_start_main+0xb4)[0x2000094843d4]
[p3dev01:32630] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node p3dev01 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
0% tests passed, 11 tests failed out of 11
Total Test time (real) = 51.72 sec
The following tests FAILED:
60 - testMath (Failed)
107 - testDruckerPrager (Failed)
109 - testModifiedCamClay (Failed)
149 - testTriaxial_druckerPragerExtended (Failed)
152 - testTriaxial_modifiedCamClay (Failed)
153 - testTriaxial_delftEggUseLinear (Failed)
154 - testTriaxial_modifiedCamClayVolumetric (Failed)
155 - testTriaxial_delftEggLoadPathDryUseLinear (Failed)
156 - testTriaxial_delftEggLoadPathWetUseLinear (Failed)
157 - testTriaxial_delftEggCase1 (Failed)
158 - testTriaxial_delftEggCase2 (Failed)
Errors while running CTest
closing this since https://github.com/GEOS-DEV/GEOS/issues/3161 is closed
