elePyant
elePyant copied to clipboard
fraserwg
elePyant (pronounced elephant) provides a set of tools for compressing netCDF files and xarray Datasets and DataArrays.
This adds a new rounding method that aims to preserve significant figures ("sig figs") in base 10 (as opposed to decimal places), and a new function `round_dataarray` that can use...
AFAIK netCDFs default compression algorithm is [zlib](https://en.wikipedia.org/wiki/Zlib). Zlib is not bad, but I highly recommend to use [Zstandard](http://facebook.github.io/zstd/) as I achieve much better compression factors with it. In general benchmarks,...
A command line interface allowing operations on NetCDF files (but not xarray objects) could be useful. I'm thinking something that would work along the lines of ```bash ncround -i /path/to/in/file.nc...
Just FYI, the unidata netCDF4 documentation contains the following section: [https://unidata.github.io/netcdf4-python/netCDF4/index.html#section9](https://unidata.github.io/netcdf4-python/netCDF4/index.html#section9) which has a similar approach to the one of elePyant.
Be aware that rounding in decimal representation does is general not round in binary representation too, as the former is not an integer power of the latter. Currently, [you seem...
A possibly dumb idea: would setting the 8 lowest mantissa bits to 0, giving all-0 bytes, compress better than *decimal* _places ? https://stackoverflow.com/questions/56961855/numpy-setting-n-last-elements-of-mantissas-in-double-array describes how to do this in numpy....
