python-lz4 icon indicating copy to clipboard operation
python-lz4 copied to clipboard

Published 20 hours ago • verified publisher icon steeve

big-endianness for size

Open facboy opened this issue 11 years ago • 2 comments

it would be great if you could specify in the method calls (loads and dumps) whether the size header was little endian or big endian.

facboy avatar Jul 09 '14 17:07 facboy

The size is always stored in little endian, to guarantee interop between archs. See https://github.com/steeve/python-lz4/blob/master/src/python-lz4.c#L44

steeve avatar Jul 09 '14 18:07 steeve

actually i'm reading data from a different library (hadoop as it happens), and its block compression uses a completely different scheme. basically it has the original size as a big-endian int32, followed by the compressed size as a big-endian int32, and then the compressed data. so a simple option wouldn't have worked in any case. maybe the ability to specify the size and then treat the data as merely the compressed stream?

facboy avatar Jul 09 '14 18:07 facboy