python-blosc
python-blosc copied to clipboard
blosc_extension.error: Error -1 while decompressing data
Some memory corruption is happening in my use case for Blosc. Blosc is being used to store numpy ndarrays in shared memory using plasma. It is unclear whether the issue is in compression/decompression or in writing/reading to shared memory. Here are my write/read methods:
def write_blosc(client: plasma.PlasmaClient, x: np.ndarray):
object_id = plasma.ObjectID(np.random.bytes(20))
buf = blosc.pack_array(x)
client.create_and_seal(object_id, buf)
return object_id
def read_blosc(client: plasma.PlasmaClient, object_id: plasma.ObjectID):
[buf] = client.get_buffers([object_id], timeout_ms=100)
res = blosc.unpack_array(buf.to_pybytes())
return res
Numpy arrays are able to be successfully read and written most of the time, but occasionally (maybe 1 call out of every 10,000 to the read function) the error in the title shows up.
(accidentally closed it)
After merging #244, this should have been fixed? At any rate, feel free to reopen if this is not the case.