WarpX
WarpX copied to clipboard
Published
20 hours ago •
ECP-WarpX
ECP-WarpX
WarpX SENSEI in-situ with VisIt got segmentation fault
Dear WarpX team, I have confirmed that WarpX can successfully run with both diagnostic format plotfile and openpmd after compiling WarpX-24.06 with SENSEI-5.0.0 and VisIt-3.4.1. However, WarpX with diagnostic sensei triggered segfault and crashed when saving the first file as follows.
Initializing AMReX (24.06)...
MPI initialized with 56 MPI processes
MPI initialized with thread support level 3
OMP initialized with 1 OMP threads
AMReX (24.06) initialized
PICSAR (23.09)
WarpX (Unknown)
__ __ __ __
\ \ / /_ _ _ __ _ __\ \/ /
\ \ /\ / / _` | '__| '_ \\ /
\ V V / (_| | | | |_) / \
\_/\_/ \__,_|_| | .__/_/\_\
|_|
Level 0: dt = 2.511099804e-16 ; dx = 4.6875e-07 ; dy = 4.6875e-07 ; dz = 7.8125e-08
SENSEI Begin initialize...
STATUS: [0][/home/u5375080/Downloads/SENSEI/5.0.0/sensei/ConfigurableAnalysis.cxx:955][v4.0.0]
STATUS: configured LibsimAnalysisAdaptor render
SENSEI initialize complete (15.8970157 sec)
Grids Summary:
Level 0 56 grids 12582912 cells 100 % of domain
smallest grid: 64 x 64 x 54 biggest grid: 64 x 64 x 55
-------------------------------------------------------------------------------
--------------------------- MAIN EM PIC PARAMETERS ----------------------------
-------------------------------------------------------------------------------
Precision: | DOUBLE
Particle precision: | DOUBLE
Geometry: | 3D (XYZ)
Operation mode: | Electromagnetic
| - vacuum
-------------------------------------------------------------------------------
Current Deposition: | Esirkepov
Particle Pusher: | Boris
Charge Deposition: | standard
Field Gathering: | energy-conserving
Particle Shape Factor:| 3
-------------------------------------------------------------------------------
Maxwell Solver: | Yee
| - staggered grid
Guard cells | - ng_alloc_EB = (4,4,4)
(allocated for E/B) |
-------------------------------------------------------------------------------
For full input parameters, see the file: warpx_used_inputs
--- INFO : Writing Sensei file diags/diag1000000
Segfault
.
.
.
See Backtrace.19.0 file for details
See Backtrace.47.0 file for details
See Backtrace.7.0 file for details
See Backtrace.35.0 file for details
See Backtrace.53.0 file for details
See Backtrace.9.0 file for details
See Backtrace.49.0 file for details
See Backtrace.21.0 file for details
See Backtrace.31.0 file for details
See Backtrace.1.0 file for details
See Backtrace.37.0 file for details
See Backtrace.39.0 file for details
See Backtrace.25.0 file for details
See Backtrace.15.0 file for details
See Backtrace.27.0 file for details
See Backtrace.45.0 file for details
See Backtrace.3.0 file for details
See Backtrace.41.0 file for details
See Backtrace.17.0 file for details
See Backtrace.13.0 file for details
See Backtrace.23.0 file for details
See Backtrace.55.0 file for details
See Backtrace.11.0 file for details
See Backtrace.43.0 file for details
See Backtrace.33.0 file for details
See Backtrace.51.0 file for details
See Backtrace.29.0 file for details
See Backtrace.5.0 file for details
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 19 in communicator MPI COMMUNICATOR 4 DUP FROM 0
with errorcode 11.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
All the libraries referenced are listed here:
[u5375080@lgn303 ~]$ ldd .local/opt/warpx/24.06/sensei/5.0.0/libsim/3.4.1/linux-x86_64_gcc-8.5/bin/warpx.3d.MPI.OMP.DP.PDP.OPMD.PSATD.EB.QED.GENQEDTABLES.SENSEI
linux-vdso.so.1 (0x00007ffc6bf59000)
libsensei.so => /home/u5375080/.local/opt/sensei/5.0.0/libsim/3.4.1/linux-x86_64_gcc-8.5/lib64/libsensei.so (0x00001485ea6c7000)
libsenseiCore.so => /home/u5375080/.local/opt/sensei/5.0.0/libsim/3.4.1/linux-x86_64_gcc-8.5/lib64/libsenseiCore.so (0x00001485ea284000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00001485ea064000)
libsvtkCommonDataModel-9.0.so.1 => /home/u5375080/.local/opt/sensei/5.0.0/libsim/3.4.1/linux-x86_64_gcc-8.5/lib64/libsvtkCommonDataModel-9.0.so.1 (0x00001485e9a25000)
libsvtkCommonTransforms-9.0.so.1 => /home/u5375080/.local/opt/sensei/5.0.0/libsim/3.4.1/linux-x86_64_gcc-8.5/lib64/libsvtkCommonTransforms-9.0.so.1 (0x00001485e97f4000)
libsvtkCommonMisc-9.0.so.1 => /home/u5375080/.local/opt/sensei/5.0.0/libsim/3.4.1/linux-x86_64_gcc-8.5/lib64/libsvtkCommonMisc-9.0.so.1 (0x00001485e95d9000)
libsvtkCommonMath-9.0.so.1 => /home/u5375080/.local/opt/sensei/5.0.0/libsim/3.4.1/linux-x86_64_gcc-8.5/lib64/libsvtkCommonMath-9.0.so.1 (0x00001485e93b8000)
libsvtkCommonCore-9.0.so.1 => /home/u5375080/.local/opt/sensei/5.0.0/libsim/3.4.1/linux-x86_64_gcc-8.5/lib64/libsvtkCommonCore-9.0.so.1 (0x00001485e8d66000)
libsvtksys-9.0.so.1 => /home/u5375080/.local/opt/sensei/5.0.0/libsim/3.4.1/linux-x86_64_gcc-8.5/lib64/libsvtksys-9.0.so.1 (0x00001485e8b1a000)
libadios2_c_mpi.so.2.10 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libadios2_c_mpi.so.2.10 (0x00001485e8916000)
libadios2_c.so.2.10 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libadios2_c.so.2.10 (0x00001485e86df000)
libadios2_cxx11_mpi.so.2.10 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libadios2_cxx11_mpi.so.2.10 (0x00001485e84d8000)
libadios2_cxx11.so.2.10 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libadios2_cxx11.so.2.10 (0x00001485e81a9000)
libvtkIOXML-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkIOXML-9.2.so.1 (0x00001485e7e98000)
libvtkIOXMLParser-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkIOXMLParser-9.2.so.1 (0x00001485e7c80000)
libvtkIOLegacy-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkIOLegacy-9.2.so.1 (0x00001485e79c9000)
libvtkIOCore-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkIOCore-9.2.so.1 (0x00001485e7763000)
libvtkRenderingCore-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkRenderingCore-9.2.so.1 (0x00001485e7337000)
libvtkFiltersGeneral-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkFiltersGeneral-9.2.so.1 (0x00001485e6d11000)
libvtkFiltersCore-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkFiltersCore-9.2.so.1 (0x00001485e6377000)
libvtkCommonExecutionModel-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkCommonExecutionModel-9.2.so.1 (0x00001485e6084000)
libvtkCommonDataModel-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkCommonDataModel-9.2.so.1 (0x00001485e596c000)
libvtkCommonTransforms-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkCommonTransforms-9.2.so.1 (0x00001485e572d000)
libvtkCommonMisc-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkCommonMisc-9.2.so.1 (0x00001485e5120000)
libvtkCommonMath-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkCommonMath-9.2.so.1 (0x00001485e4efc000)
libvtkkissfft-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkkissfft-9.2.so.1 (0x00001485e4cf5000)
libvtkCommonCore-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkCommonCore-9.2.so.1 (0x00001485e437b000)
libvtksys-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtksys-9.2.so.1 (0x00001485e412e000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00001485e3f2a000)
_PythonAnalysis.so => /home/u5375080/.local/opt/sensei/5.0.0/libsim/3.4.1/linux-x86_64_gcc-8.5/lib64/python3.9/site-packages/sensei/_PythonAnalysis.so (0x00001485e3b57000)
libfftw3.so.3 => /home/u5375080/.local/opt/visit/3.4.1/third_party/fftw/3.3.10/linux-x86_64_gcc-8.5/lib/libfftw3.so.3 (0x00001485e384b000)
libfftw3_omp.so.3 => /home/u5375080/.local/opt/visit/3.4.1/third_party/fftw/3.3.10/linux-x86_64_gcc-8.5/lib/libfftw3_omp.so.3 (0x00001485e3644000)
libmpi_cxx.so.40 => /home/u5375080/.local/opt/visit/3.4.1/third_party/openmpi/4.1.6/linux-x86_64_gcc-8.5/lib/libmpi_cxx.so.40 (0x00001485e3429000)
libhdf5_mpi.so.9 => /home/u5375080/.local/opt/visit/3.4.1/third_party/hdf5_mpi/1.8.14/linux-x86_64_gcc-8.5/lib/libhdf5_mpi.so.9 (0x00001485e2f2c000)
libsz.so.2 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libsz.so.2 (0x00001485e2d18000)
libz.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libz.so.1 (0x00001485e2b01000)
libmpi.so.40 => /home/u5375080/.local/opt/visit/3.4.1/third_party/openmpi/4.1.6/linux-x86_64_gcc-8.5/lib/libmpi.so.40 (0x00001485e27d6000)
libgomp.so.1 => /usr/lib64/libgomp.so.1 (0x00001485e259e000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00001485e2209000)
libm.so.6 => /usr/lib64/libm.so.6 (0x00001485e1e87000)
libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00001485e1c6f000)
libc.so.6 => /usr/lib64/libc.so.6 (0x00001485e18aa000)
/lib64/ld-linux-x86-64.so.2 (0x00001485ea909000)
libsvtkCommonSystem-9.0.so.1 => /home/u5375080/.local/opt/sensei/5.0.0/libsim/3.4.1/linux-x86_64_gcc-8.5/lib64/libsvtkCommonSystem-9.0.so.1 (0x00001485e1695000)
libadios2_core_mpi.so.2.10 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libadios2_core_mpi.so.2.10 (0x00001485e141d000)
libadios2_core.so.2.10 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libadios2_core.so.2.10 (0x00001485e0ae0000)
libvtkCommonSystem-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkCommonSystem-9.2.so.1 (0x00001485e08a2000)
libvtkexpat-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkexpat-9.2.so.1 (0x00001485e0661000)
libvtkdoubleconversion-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkdoubleconversion-9.2.so.1 (0x00001485e044f000)
libvtklz4-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtklz4-9.2.so.1 (0x00001485e021c000)
libvtklzma-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtklzma-9.2.so.1 (0x00001485dffee000)
libvtkCommonColor-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkCommonColor-9.2.so.1 (0x00001485dfdd1000)
libvtkFiltersGeometry-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkFiltersGeometry-9.2.so.1 (0x00001485dfae2000)
libvtkFiltersSources-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkFiltersSources-9.2.so.1 (0x00001485df802000)
libvtkCommonComputationalGeometry-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkCommonComputationalGeometry-9.2.so.1 (0x00001485df5cd000)
libvtkfmt-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkfmt-9.2.so.1 (0x00001485df3af000)
libvtkpugixml-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkpugixml-9.2.so.1 (0x00001485df179000)
libvtkloguru-9.2.so.1 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libvtkloguru-9.2.so.1 (0x00001485def45000)
_pysvtk.so => /home/u5375080/.local/opt/sensei/5.0.0/libsim/3.4.1/linux-x86_64_gcc-8.5/lib64/python3.9/site-packages/sensei/../svtk/_pysvtk.so (0x00001485dea0b000)
libpython3.9.so.1.0 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libpython3.9.so.1.0 (0x00001485de4a0000)
libopen-rte.so.40 => /opt/ohpc/Taiwania3/Intel/lib/gcc-8.5.0/openmpi-4.1.6/lib64/libopen-rte.so.40 (0x00001485de1e9000)
libopen-pal.so.40 => /opt/ohpc/Taiwania3/Intel/lib/gcc-8.5.0/openmpi-4.1.6/lib64/libopen-pal.so.40 (0x00001485ddf3b000)
librt.so.1 => /lib64/librt.so.1 (0x00001485ddd33000)
libutil.so.1 => /lib64/libutil.so.1 (0x00001485ddb2f000)
libhwloc.so.15 => /lib64/libhwloc.so.15 (0x00001485dd8df000)
libevent_core-2.1.so.6 => /lib64/libevent_core-2.1.so.6 (0x00001485dd6a6000)
libevent_pthreads-2.1.so.6 => /lib64/libevent_pthreads-2.1.so.6 (0x00001485dd4a3000)
libadios2_perfstubs.so.2.10 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libadios2_perfstubs.so.2.10 (0x00001485dd29f000)
libbz2.so.1 => /usr/lib64/libbz2.so.1 (0x00001485dd08e000)
libpng16.so.16 => /usr/lib64/libpng16.so.16 (0x00001485dce59000)
libadios2_evpath.so.2.10 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libadios2_evpath.so.2.10 (0x00001485dcbe6000)
libadios2_ffs.so.2.10 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libadios2_ffs.so.2.10 (0x00001485dc987000)
libadios2_atl.so.2.10 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libadios2_atl.so.2.10 (0x00001485dc77b000)
libfabric.so.1 => /opt/ohpc/Taiwania3/Intel/lib/libfabric/1.20.0/lib64/libfabric.so.1 (0x00001485dbd12000)
libucp.so.0 => /usr/lib64/libucp.so.0 (0x00001485dba4f000)
libuct.so.0 => /usr/lib64/libuct.so.0 (0x00001485db815000)
libucs.so.0 => /usr/lib64/libucs.so.0 (0x00001485db475000)
libucm.so.0 => /usr/lib64/libucm.so.0 (0x00001485db25b000)
libblosc2.so.2 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libblosc2.so.2 (0x00001485dae9e000)
libhdf5.so.9 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libhdf5.so.9 (0x00001485da9bd000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00001485da794000)
libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00001485da2ab000)
libadios2_dill.so.2.10 => /home/u5375080/.local/opt/visit/3.4.1/linux-x86_64_gcc-8.5/3.4.1/linux-x86_64/lib/libadios2_dill.so.2.10 (0x00001485da05a000)
librdmacm.so.1 => /lib64/librdmacm.so.1 (0x00001485d9e3f000)
libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00001485d9c1c000)
libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x00001485d9996000)
libibverbs.so.1 => /lib64/libibverbs.so.1 (0x00001485d9776000)
libuuid.so.1 => /lib64/libuuid.so.1 (0x00001485d956e000)
libnuma.so.1 => /lib64/libnuma.so.1 (0x00001485d9362000)
libatomic.so.1 => /lib64/libatomic.so.1 (0x00001485d915a000)
libxpmem.so.0 => /opt/ohpc/Taiwania3/Intel/lib/xpmem/0.2/lib64/libxpmem.so.0 (0x00001485d8f58000)
libffi.so.6 => /usr/lib64/libffi.so.6 (0x00001485d8d4f000)
The used input files for WarpX, SENSEI, VisIt, and SLURM can be found here: Input.zip
The backtraces can be downloaded here: Backtraces.zip
Best regards, Jeff
Thank you for the report and details!
I am not sure if @burlen can give us more pointers on this, I have personally not yet worked with the libSim adapter of VisIt through SENSEI :)
