elePyant icon indicating copy to clipboard operation
elePyant copied to clipboard

Published 20 hours ago • verified publisher icon fraserwg

Zstandard as default?

Open milankl opened this issue 4 years ago • 3 comments

AFAIK netCDFs default compression algorithm is zlib. Zlib is not bad, but I highly recommend to use Zstandard as I achieve much better compression factors with it. In general benchmarks, Zstd is better than Zlib in every aspect. Could you check whether there is a way that the netCDF python implementation supports it?

milankl avatar Sep 17 '20 11:09 milankl

Typical compression factors I achieved with Zstandard are about 2-3x higher than with Zlib.

milankl avatar Sep 17 '20 11:09 milankl

@milankl It seems like netcdf4-python only officially supports (de)compressing with zlib. But xarray's h5netcdf engine could maybe do it with these h5py extensions.

zmoon avatar Oct 02 '20 21:10 zmoon

Interesting, yeah, I realised that netCDF is always behind the functionality that hdf5 offers. So I guess people want to standardisation of netCDF but the features of HDF5, which puts us somewhere in between...

milankl avatar Oct 03 '20 13:10 milankl