Unidata
filter plugin path with nccopy
have a question about getting netcdf to find my filter plugin. I am trying to use a self-installed hdf5 plugin with hdf5 and netcdf installed on the NCAR system (casper).
For example, for zfp, I installed H5Z-ZFP-1.1.1( with the HDF5_ROOT set to a system-installed hdf5 (1.12.2). I then sent HDF5_PLUGIN_PATH to the location of the zfp plugin that I installed myself.
echo $HDF5_PLUGIN_PATH /glade/work/abaker/compress-libs/zfp_plugin/plugin
But when I use the nccopy command with the zfp filter it can't find it:
nccopy -F TS,32013,3,0,1202590843,1065646817 TS.nc TS_zfp_a1e-2.nc NetCDF: Filter error: undefined filter encountered Location: file nccopy.c; fcn copy_var_specials line 1175
And when I check nc-config for the plugin dir:
nc-config --plugindir /glade/u/apps/casper/23.10/spack/opt/spack/netcdf/4.9.2/packages/netcdf-c/4.9.2/gcc/12.2.0/obej/plugins
But I can't copy my plugin to the system directory, so how can I specify nccopy to use the $HDF5_PLUGIN_PATH?
Thanks!
I'm guessing that you are using netCDF 4.9.2. I'm reviewing a PR, #3034 and I wonder if you might be able to either test using the his branch, or possibly even with netCDF-C 4.9.3-rc1. @DennisHeimbigner, does any other solution for v4.9.2 leap out at you?
Lets start with some information. Can you do this command and send be the output as text?
ncdump -h TS.nc
Second, you should have files called something like H5Zzfp.c and (maybe) H5Zzfp.h. Can you send me those files? You can mail them directly to me at [email protected]
netcdf TS { dimensions: time = UNLIMITED ; // (60 currently) lat = 192 ; lon = 288 ; chars = 8 ; ilev = 31 ; lev = 30 ; slat = 191 ; slon = 288 ; nbnd = 2 ; variables: double P0 ; P0:long_name = "reference pressure" ; P0:units = "Pa" ; float TS(time, lat, lon) ; TS:units = "K" ; TS:long_name = "Surface temperature (radiative)" ; TS:cell_methods = "time: mean" ; double ch4vmr(time) ; ch4vmr:long_name = "ch4 volume mixing ratio" ; double co2vmr(time) ; co2vmr:long_name = "co2 volume mixing ratio" ; int date(time) ; date:long_name = "current date (YYYYMMDD)" ; char date_written(time, chars) ; int datesec(time) ; datesec:long_name = "current seconds of current date" ; double f11vmr(time) ; f11vmr:long_name = "f11 volume mixing ratio" ; double f12vmr(time) ; f12vmr:long_name = "f12 volume mixing ratio" ; double gw(lat) ; gw:long_name = "gauss weights" ; double hyai(ilev) ; hyai:long_name = "hybrid A coefficient at layer interfaces" ; double hyam(lev) ; hyam:long_name = "hybrid A coefficient at layer midpoints" ; double hybi(ilev) ; hybi:long_name = "hybrid B coefficient at layer interfaces" ; double hybm(lev) ; hybm:long_name = "hybrid B coefficient at layer midpoints" ; double ilev(ilev) ; ilev:long_name = "hybrid level at interfaces (1000*(A+B))" ; ilev:units = "level" ; ilev:positive = "down" ; ilev:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate" ; ilev:formula_terms = "a: hyai b: hybi p0: P0 ps: PS" ; double lat(lat) ; lat:long_name = "latitude" ; lat:units = "degrees_north" ; double lev(lev) ; lev:long_name = "hybrid level at midpoints (1000*(A+B))" ; lev:units = "level" ; lev:positive = "down" ; lev:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate" ; lev:formula_terms = "a: hyam b: hybm p0: P0 ps: PS" ; double lon(lon) ; lon:long_name = "longitude" ; lon:units = "degrees_east" ; int mdt ; mdt:long_name = "timestep" ; mdt:units = "s" ; double n2ovmr(time) ; n2ovmr:long_name = "n2o volume mixing ratio" ; int nbdate ; nbdate:long_name = "base date (YYYYMMDD)" ; int nbsec ; nbsec:long_name = "seconds of base date" ; int ndbase ; ndbase:long_name = "base day" ; int ndcur(time) ; ndcur:long_name = "current day (from base day)" ; int nlon(lat) ; nlon:long_name = "number of longitudes" ; int nsbase ; nsbase:long_name = "seconds of base day" ; int nscur(time) ; nscur:long_name = "current seconds of current day" ; int nsteph(time) ; nsteph:long_name = "current timestep" ; int ntrk ; ntrk:long_name = "spectral truncation parameter K" ; int ntrm ; ntrm:long_name = "spectral truncation parameter M" ; int ntrn ; ntrn:long_name = "spectral truncation parameter N" ; double slat(slat) ; slat:long_name = "staggered latitude" ; slat:units = "degrees_north" ; double slon(slon) ; slon:long_name = "staggered longitude" ; slon:units = "degrees_east" ; double sol_tsi(time) ; sol_tsi:long_name = "total solar irradiance" ; sol_tsi:units = "W/m2" ; double time(time) ; time:long_name = "time" ; time:units = "days since 2006-01-01 00:00:00" ; time:calendar = "noleap" ; time:bounds = "time_bnds" ; double time_bnds(time, nbnd) ; time_bnds:long_name = "time interval endpoints" ; char time_written(time, chars) ; double w_stag(slat) ; w_stag:long_name = "staggered latitude weights" ; int wnummax(lat) ; wnummax:long_name = "cutoff Fourier wavenumber" ;
// global attributes: :Conventions = "CF-1.0" ; :source = "CAM" ; :case = "b.e11.BRCP85C5CNBDRD.f09_g16.031" ; :title = "UNSET" ; :logname = "mickelso" ; :host = "ys1023" ; :Version = "$Name$" ; :revision_Id = "$Id$" ; :initial_file = "b.e11.B20TRC5CNBDRD.f09_g16.031.cam.i.2006-01-01-00000.nc" ; :topography_file = "/glade/p/cesmdata/cseg/inputdata/atm/cam/topo/USGS-gtopo30_0.9x1.25_remap_c051027. nc" ; :history = "Mon May 10 15:27:00 2021: ncks -d time,0,59,1 b.e11.BRCP85C5CNBDRD.f09_g16.031.cam.h0.TS.2 00601-208012.nc b.e11.BRCP85C5CNBDRD.f09_g16.031.cam.h0.TS.200601-2007012.nc" ; :NCO = "netCDF Operators version 4.9.5 (Homepage = http://nco.sf.net, Code = http://github.com/nco/nco )" ; }
Look into your HDF5_PLUGIN_PATH directory (i.e. /glade/work/abaker/compress-libs/zfp_plugin/plugin). Look for all files matching the pattern 'zfp.so' Please tell me what it shows.
abaker@casper-login2:~> cd $HDF5_PLUGIN_PATH abaker@casper-login2:/glade/work/abaker/compress-libs/zfp_plugin/plugin> ls libh5z-sperr.so* libh5zzfp.so*
Ok, it appears you are using the wrong value for HDF5_PLUGIN_PATH. I suspect that you need to use "/glade/work/abaker/compress-libs/zfp_plugin/plugin/.libs". You can verify by trying this command:
find /glade/work/abaker/compress-libs/zfp_plugin/plugin -name '*zfp*'
abaker@casper-login2:/glade/work/abaker/compress-libs/zfp_plugin/plugin> ls -a ./ ../ libh5z-sperr.so* libh5zzfp.so* abaker@casper-login2:/glade/work/abaker/compress-libs/zfp_plugin/plugin> find /glade/work/abaker/compress-libs/zfp_plugin/plugin -name 'zfp' /glade/work/abaker/compress-libs/zfp_plugin/plugin/libh5zzfp.so
The naming is wrong: there should be a file named 'lib__nch5zfp.so' in the '/glade/work/abaker/compress-libs/zfp_plugin/plugin/' directory
Can you tell where the 'libh5zzfp.so' is coming from? Can you show the fragment of code in some Makefile.am or CMakeLists.txt?
Thanks for you help. I just followed the cmake instructions on https://h5z-zfp.readthedocs.io/en/latest/installation.html, which indicates that it will install the following:
$(PREFIX)/include/{H5Zzfp.h,H5Zzfp_plugin.h,H5Zzfp_props.h,H5Zzfp_lib.h} $(PREFIX)/plugin/libh5zzfp.{so,dylib} $(PREFIX)/lib/libh5zzfp.a
which is what I have. The documentation also says:
"To use the installed filter as an HDF5 plugin, you would specify, for example, setenv HDF5_PLUGIN_PATH $(PREFIX)/plugin"
Here is what I did: abaker@casper-login2:/glade/work/abaker/H5Z-ZFP-1.1.1> mkdir build export ZFP_DIR=/glade/work/abaker/zfp-1.0.1/build export HDF5_DIR=/glade/u/apps/casper/23.10/spack/opt/spack/hdf5/1.12.2/gcc/12.2.0/hsod
(These below are already set: CC=gcc FC=gfortran )
cd build cmake -DBUILD_TESTING=ON -DCMAKE_INSTALL_PREFIX=/glade/work/abaker/compress-libs/zfp_plugin .. make make install
The only modification I made was to add the link for the math libraries in CMakeLIsts.txt: target_link_libraries(test_write_plugin m h5z_zfp_shared) (https://github.com/LLNL/H5Z-ZFP/blob/master/CMakeLists.txt)
Do you want my CMakeCache.txt in my build directory? (If you have access to glade, my files are readable).
Can you send me the file "H5Zzfp_props_private.h"`; I am trying to build.