netcdf-c
netcdf-c copied to clipboard
Zarr copy fails on write - variable uncompressable
When experimenting with #3068 I bumped into this. I believe it was not introduced in such PR and trying with 4.8.1 it works fine.
nccopy file://ngc4008_P1D_0_crs.zarr.zip#mode=zarr,zip file:///var/tmp/out.zarr#mode=zarr,file
Blosc_Filter Error: blosc_filter: Buffer is uncompressible.
NetCDF: Filter error: bad id or parameters or duplicate filter
Location: file /home/k/k202186/repos/netcdf-c/ncdump/nccopy.c; fcn copy line 2138
I pin pointed the variable that was causing the error and created a simplied dataset that could reproduce it: ngc4008_P1D_0_crs.zarr.zip
the respective ncdump works without errors:
$ ncdump file:///home/k/k202186/ngc4008_P1D_0_crs.zarr#mode=zarr,file
netcdf ngc4008_P1D_0_crs {
dimensions:
crs = 1 ;
variables:
float crs(crs) ;
crs:grid_mapping_name = "healpix" ;
crs:healpix_nside = 1 ;
crs:healpix_order = "nest" ;
data:
crs = NaNf ;
}
I'll try to follow it up.
Relevant logs
NCZ_close: ncid 0x20000
ncz_closeorabort: file: 0x625d3453acf0
ncz_sync_netcdf4_file
*** NetCDF-4 Internal Metadata: int_ncid 0x0 ext_ncid 0x20000
FILE - path: file:///var/tmp/ngc4008_P1D_0.zarr#mode=zarr,file,nczarr cmode: 0x1001 parallel: 0 redef: 0 fill_mode: 256 no_write: 0 next_nc_grpid: 1
GROUP - / nc_grpid: 0 nvars: 1 natts: 1
GROUP ATTRIBUTE - attnum: 0 name: _NCProperties type: 2 len: 47
DIMENSION - dimid: 0 name: crs len: 1 unlimited: 0
VARIABLE - varid: 0 name: crs ndims: 1 dimids: 0 storage: chunked
VAR ATTRIBUTE - attnum: 0 name: grid_mapping_name type: 2 len: 7
VAR ATTRIBUTE - attnum: 1 name: healpix_nside type: 4 len: 1
VAR ATTRIBUTE - attnum: 2 name: healpix_order type: 2 len: 4
NCZ_write_ncproperties
ncz_sync_file: file: file:///var/tmp/ngc4008_P1D_0.zarr#mode=zarr,file,nczarr
ncz_sync_grp: dims: (null)
ncz_sync_atts
NCZ_inq_var_all: ncid 0x20000 varid 0
ncz_find_grp_var_att: ncid 131072 varid 0 attnum 0 use_name 0
NC4_inq_var_all: ncid 0x20000 varid 0
NCZ_inq_var_all: ncid 0x20000 varid 0
ncz_find_grp_var_att: ncid 131072 varid 0 attnum 0 use_name 0
NC4_inq_var_all: ncid 0x20000 varid 0
chunksizesp[0]=1
ncz_sync_atts
put_chunk: var: 0x625d3453fa30
Blosc_Filter Error: blosc_filter: Buffer is uncompressible.
NetCDF: Filter error: bad id or parameters or duplicate filter
Location: file /home/reis/repos/netcdf-c/ncdump/nccopy.c; fcn copy line 2138
NC4_clear_provenance
NCZ_clear_provenance
The failure is on the compression side. As you note, ncdump works because is doing decompression only. In looking at the cblosc doc. it appears that there is a simple fix, so I modified plugins/H5Zblosc.c to include that fix.