geodynamics
netcdf runtime and postprocessing errors
Hi all,
Attached is a benchmark of AxiSEM synthetics against FK synthetics, using the "scak" model from the University of Alaska, Fairbanks. The fits look pretty good!
Getting this result took a lot of debugging. Since AxiSEM is such a useful package for us, if anyone has any advice regarding the following, it'd be really helpful to know.
Thanks, Ryan
-
on our system, instaseis was unable to read netcdf files processed using the field_transform.sh script
-
instaseis successfully read netcdf files processed by the field_transform.py script. However, despite trying a variety block_sizes and cache_sizes, the processing was extremely slow (<< 1 MB/s), which led to processing times of several days for a 2 s dominant period simulation
-
in short simulations, the AxiSEM solver and mesher ran reliably, but in longer runs or runs with nproc > 4, fatal netcdf runtime errors were a frequent occurrence
-
when LOCAL_MAX_COLAT in MESHER/inparm_mesh was uncommented, instaseis was unable to read the resulting netcdf output files
-
to compile AxiSEM in debugging mode on our cluster, we had to add -pthread to the gfortran compiler flags in make_axisem.macros
Dear Ryan, thanks for reporting, some thoughts:
on our system, instaseis was unable to read netcdf files processed using the field_transform.sh script
This seems to be an inconsistency in libraries (fortran netcdf vs h5netcdf) that has been around for a while now.
instaseis successfully read netcdf files processed by the field_transform.py script. However, despite trying a variety block_sizes and cache_sizes, the processing was extremely slow (<< 1 MB/s), which led to processing times of several days for a 2 s dominant period simulation
This was a quick hack to solve the first issue, by now I would suggest to use the more versatile tool included in instaseis for repacking (http://instaseis.net/database_repacking.html#repacking-script).
in short simulations, the AxiSEM solver and mesher ran reliably, but in longer runs or runs with nproc > 4, fatal netcdf runtime errors were a frequent occurrence
to compile AxiSEM in debugging mode on our cluster, we had to add -pthread to the gfortran compiler flags in make_axisem.macros
This sounds to me like you are not using the parallel netcdf library, but our own round robin parallelization with threads. Maybe try the parallel netcdf instead (USE_PAR_NETCDF = True in make_axisem.macros)
when LOCAL_MAX_COLAT in MESHER/inparm_mesh was uncommented, instaseis was unable to read the resulting netcdf output files
I think I have been using this feature with instaseis before, but it is definitely not well tested. If you can't get around it, I would need a minimal example to reproduce.
Martin
