Alternative data types for output?

[jamestwebber]

This crate is super useful, thanks for writing/maintaining it!

I don't know if this is feasible but it would be nice if this crate could output to different types beyond u32. I needed to output to u8 to save space and I ended up rewriting a lot stuff for my purposes (I am only decompressing and always have the same width so I could simplify things).

Having done so, I think that it'd be possible to add another level of macro magic here and output to u8 and u16 for when those types are needed. I'm not too familiar with programming rust macros though, it might create a huge binary if it wasn't done carefully.

Just thought I'd put it out there as a potential enhancement, thanks again for the crate.

[fulmicoton]

This is not useful for tantivy so I won't implement it, but I accept pull request.

The binary size is indeed a problem, but we can put it behind a compilation flag.

[jean343]

I would like to see support for u64 for large databases.

[fulmicoton]

@jean343 same answer as above unfortunately...

[jean343]

Thank you for the answer. While I think about it, we are only storing deltas, and most u64 deltas can be stored in u32. So this isn't as big of a limitation as I thought. Thanks again for your awesome library.

[jorgecarleitao]

I will try to take a stab a this for the Bitpack1.

The parquet format uses this format and accepts both u32 and u64.

[jorgecarleitao]

Not very easy in stable rust: the support for const generics is limited and we need it to call simd instructions irrespectively of the type, which requires a constant generic. Alternative is via macros. Will keep digging.