snappy-java icon indicating copy to clipboard operation
snappy-java copied to clipboard

BitShuffle#shuffle does not allow to pass arbitrary type sizes (even if the underlying library does support that)

Open foosmate opened this issue 4 years ago • 1 comments

For data encoding, sometimes integers with intermediate sizes (e.g. of 3 or 5 bytes) are used (e.g. to reduce bandwith). These arrays currently cannot be passed to the external BitShuffle library with snappy-java, because in snappy-java the typesize is predefined as constant. Since the external library actually supports arbitrary type sizes this is an unnecessary restriction.

How to fix:

Would be great to have the following methods:

public static int shuffle(ByteBuffer input, int typeSize, ByteBuffer shuffled) throws IOException {
    ...
}

public static int unshuffle(ByteBuffer shuffled, int typeSize, ByteBuffer output) throws IOException {
    ...
}

foosmate avatar Nov 10 '21 15:11 foosmate

Related #443

xerial avatar May 23 '23 22:05 xerial