bcolz
bcolz copied to clipboard
Runtime and Memory errors when moving from python 2.7 to python 3.6
Recently, I migrated my python environment to 3.6 and updated bcolz along with it.
Now I have trouble opening bcolz containers I created with the python 2.7 environment.
I get
Exception ignored in: 'bcolz.carray_ext.chunk._getitem' RuntimeError: fatal error during Blosc decompression: -1
and the data I get is all zeros. I noticed that a lot of changes have been made to the compression algorithms which you call with the "cname" argument.
I used to use
with bcolz.defaults_ctx(vm='python', cparams=bcolz.cparams(clevel=5, cname='zstd')):
These settings are apparently not offered any more accourding to the manual on page 11.
Would it be possible to include the old algorithms for backward compatibility?
Moreover, I noticed that the same data which used to fit into my RAM just about now raises a memory error. This could be related to changing the compression algorithms, too. However, I am a bit at a loss on when to use which algorithm (cname) with which compression level (clevel). It would be very helpful if you could publish comprehensive benchmarks or guidelines on when to use which setting.
For example, which settings would be the closest match to the above ones?