Castro icon indicating copy to clipboard operation
Castro copied to clipboard

XRB with tracer particles fails with 2 levels of refinement (ref_ratio of 4 then 2)

Open nmford20 opened this issue 3 years ago • 0 comments

(on Cori Haswell nodes, using the 3e8 K setup) The tracer particles implementation with the XRB flame wave simulations fails in the case of 2 refinement levels, ref_ratio of 4 then 2. It does not fail for 0, 1 refinement levels, and works for 2 levels if the ref_ratio is 2 and then 4, so the issue seems to be specifically in setting the first refinement factor (level 0 to level 1) to 4. It seems to be an error related to the AMReX ParticleContainer, with the backtrace indicating it's an issue in Redistribute after Castro regrids. The Backtrace shows:

3: /global/project/projectdirs/m3018/XRB/flame_wave_particles/test7_2ref/./Castro2d.gnu.haswell.MPI.OMP.ex() [0x20060f7b] _ZN5amrex17ParticleContainerILi2ELi0ELi0ELi0ESaE15RedistributeCPUEiiii._omp_fn.2 ../../../external/amrex/Src/Particle/AMReX_ParticleContainerI.H:1270:19

4: /opt/gcc/8.3.0/snos/lib64/libgomp.so.1(GOMP_parallel+0x3f) [0x2aaaacccfb0f] GOMP_parallel ../../../cray-gcc-8.3.0-201903122028.16ea96cb84a9a/libgomp/parallel.c:170:1

5: /global/project/projectdirs/m3018/XRB/flame_wave_particles/test7_2ref/./Castro2d.gnu.haswell.MPI.OMP.ex() [0x200679ae] _ZNSt6vectorIPN5amrex12ParticleTileILi2ELi0ELi0ELi0ESaEESaIS3_EED4Ev inlined at ../../../external/amrex/Src/Base/AMReX_Vector.H:25:7 in _ZN5amrex17ParticleContainerILi2ELi0ELi0ELi0ESaE15RedistributeCPUEiiii /opt/gcc/8.3.0/snos/include/g++/bits/stl_vector.h:567:15 _ZN5amrex6VectorIPNS_12ParticleTileILi2ELi0ELi0ELi0ESaEESaIS3_EED4Ev ../../../external/amrex/Src/Base/AMReX_Vector.H:25:7 _ZN5amrex17ParticleContainerILi2ELi0ELi0ELi0ESaE15RedistributeCPUEiiii ../../../external/amrex/Src/Particle/AMReX_ParticleContainerI.H:1227:33

6: /global/project/projectdirs/m3018/XRB/flame_wave_particles/test7_2ref/./Castro2d.gnu.haswell.MPI.OMP.ex() [0x20049e62] _ZN5amrex17ParticleContainerILi2ELi0ELi0ELi0ESaE12RedistributeEiiii inlined at ../../../Source/driver/Castro.cpp:2217:31 in _ZN6Castro11post_regridEii ../../../external/amrex/Src/Particle/AMReX_ParticleContainerI.H:929:5 _ZN6Castro11post_regridEii

nmford20 avatar Apr 09 '21 05:04 nmford20