YAXArrays.jl icon indicating copy to clipboard operation
YAXArrays.jl copied to clipboard

savecube does not take backend kwargs anymore

Open felixcremer opened this issue 2 years ago • 4 comments

Before #150 it was possible to give backend kwargs to example set the compression. This seems to be not possible anymore.

We would need to find a good way to also provide documentation about the available kwargs for different backends.

felixcremer avatar Jan 13 '23 09:01 felixcremer

This is fixed by #267. We still don't have a good way to expose the backend documentation.

felixcremer avatar Jul 24 '23 08:07 felixcremer

Hello!

Is there an example on how to set compression? My output gives a ~ 35-40GB dataset and I'd like to compress it as much as possible. Thanks!

Balinus avatar Jan 16 '24 22:01 Balinus

No unfortunately there isn't and we should add one to the docs. It depends on the backend how you have to specify the compression. When you are using NetCDF you can use the compress keyword to the savecube function where you can set the compression level between 0 and 9 as Integers. Thereby 0 is no compression and 9 is the highest compression which would then take the longest for writing the data.

For Zarr one can set a compressor keyword. The compressor can be set as a Zarr.BloscCompressor(clevel=n) where n is the compression level with the same meaning as in the NetCDF case. See the docs here: https://juliaio.github.io/Zarr.jl/latest/reference/#Compressors

Maybe we could streamline this and have an YAXArrays internal compress keyword which would then be converted to the BloscCompressor with the correct clevel.

felixcremer avatar Jan 17 '24 11:01 felixcremer

Thanks! I will try it and add it to the documentation

Balinus avatar Jan 17 '24 17:01 Balinus

#438

Balinus avatar Sep 13 '24 17:09 Balinus