python-blosc icon indicating copy to clipboard operation
python-blosc copied to clipboard

blosc_extension.error: Error -1 while decompressing data

Open nikhilshinday opened this issue 6 years ago • 1 comments

Some memory corruption is happening in my use case for Blosc. Blosc is being used to store numpy ndarrays in shared memory using plasma. It is unclear whether the issue is in compression/decompression or in writing/reading to shared memory. Here are my write/read methods:


def write_blosc(client: plasma.PlasmaClient, x: np.ndarray):
    object_id = plasma.ObjectID(np.random.bytes(20))
    buf = blosc.pack_array(x)
    client.create_and_seal(object_id, buf)
    return object_id


def read_blosc(client: plasma.PlasmaClient, object_id: plasma.ObjectID):
    [buf] = client.get_buffers([object_id], timeout_ms=100)
    res = blosc.unpack_array(buf.to_pybytes())
    return res

Numpy arrays are able to be successfully read and written most of the time, but occasionally (maybe 1 call out of every 10,000 to the read function) the error in the title shows up.

nikhilshinday avatar Dec 18 '19 22:12 nikhilshinday

(accidentally closed it)

nikhilshinday avatar Dec 18 '19 22:12 nikhilshinday

After merging #244, this should have been fixed? At any rate, feel free to reopen if this is not the case.

FrancescAlted avatar Dec 07 '22 17:12 FrancescAlted