netcdf-c icon indicating copy to clipboard operation
netcdf-c copied to clipboard

Zarr copy fails on write - variable uncompressable

Open mannreis opened this issue 11 months ago • 1 comments

When experimenting with #3068 I bumped into this. I believe it was not introduced in such PR and trying with 4.8.1 it works fine.

nccopy file://ngc4008_P1D_0_crs.zarr.zip#mode=zarr,zip  file:///var/tmp/out.zarr#mode=zarr,file
Blosc_Filter Error: blosc_filter: Buffer is uncompressible.
NetCDF: Filter error: bad id or parameters or duplicate filter
Location: file /home/k/k202186/repos/netcdf-c/ncdump/nccopy.c; fcn copy line 2138

I pin pointed the variable that was causing the error and created a simplied dataset that could reproduce it: ngc4008_P1D_0_crs.zarr.zip

the respective ncdump works without errors:

$ ncdump file:///home/k/k202186/ngc4008_P1D_0_crs.zarr#mode=zarr,file
netcdf ngc4008_P1D_0_crs {
dimensions:
	crs = 1 ;
variables:
	float crs(crs) ;
		crs:grid_mapping_name = "healpix" ;
		crs:healpix_nside = 1 ;
		crs:healpix_order = "nest" ;
data:

 crs = NaNf ;
}

I'll try to follow it up.


Relevant logs

NCZ_close: ncid 0x20000
		ncz_closeorabort: file: 0x625d3453acf0
			ncz_sync_netcdf4_file
		*** NetCDF-4 Internal Metadata: int_ncid 0x0 ext_ncid 0x20000
		FILE - path: file:///var/tmp/ngc4008_P1D_0.zarr#mode=zarr,file,nczarr cmode: 0x1001 parallel: 0 redef: 0 fill_mode: 256 no_write: 0 next_nc_grpid: 1
		 GROUP - / nc_grpid: 0 nvars: 1 natts: 1
		 GROUP ATTRIBUTE - attnum: 0 name: _NCProperties type: 2 len: 47
		 DIMENSION - dimid: 0 name: crs len: 1 unlimited: 0
		 VARIABLE - varid: 0 name: crs ndims: 1 dimids: 0 storage: chunked
		 VAR ATTRIBUTE - attnum: 0 name: grid_mapping_name type: 2 len: 7
		 VAR ATTRIBUTE - attnum: 1 name: healpix_nside type: 4 len: 1
		 VAR ATTRIBUTE - attnum: 2 name: healpix_order type: 2 len: 4
					NCZ_write_ncproperties
			ncz_sync_file: file: file:///var/tmp/ngc4008_P1D_0.zarr#mode=zarr,file,nczarr
			ncz_sync_grp: dims: (null)
			ncz_sync_atts
		NCZ_inq_var_all: ncid 0x20000 varid 0
				ncz_find_grp_var_att: ncid 131072 varid 0 attnum 0 use_name 0
		NC4_inq_var_all: ncid 0x20000 varid 0
		NCZ_inq_var_all: ncid 0x20000 varid 0
				ncz_find_grp_var_att: ncid 131072 varid 0 attnum 0 use_name 0
		NC4_inq_var_all: ncid 0x20000 varid 0
				chunksizesp[0]=1
			ncz_sync_atts
			put_chunk: var: 0x625d3453fa30
Blosc_Filter Error: blosc_filter: Buffer is uncompressible.
NetCDF: Filter error: bad id or parameters or duplicate filter
Location: file /home/reis/repos/netcdf-c/ncdump/nccopy.c; fcn copy line 2138
					NC4_clear_provenance
					NCZ_clear_provenance

mannreis avatar Jan 14 '25 13:01 mannreis

The failure is on the compression side. As you note, ncdump works because is doing decompression only. In looking at the cblosc doc. it appears that there is a simple fix, so I modified plugins/H5Zblosc.c to include that fix.

DennisHeimbigner avatar Jan 15 '25 23:01 DennisHeimbigner