gyscos
Please support seekable format
There's a seekable zstd format, which makes it possible to seek to and decompress segments independently. It includes a jump table at the end for all the segments.
https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md
I'd love to have support for this format in zstd-rs, both to compress (with an API to mark the end of a segment) and to decompress (parsing the jump table and allowing decompression of individual segments).
There's a separate binding crate https://lib.rs/crates/zstd-seekable which can be used as a reference implementation
cc @gyscos Are you interested in having this seekable zstd in this repository/crate?
Looking at the code, there's a ZSTD_seekable and ZSTD_seekable_compressor, so it looks like just two more bindings to add.
If it is just binding, then I could give it a shot.
I've been interested in this issue. I originally tried using zstd-seekable, but it only works with sync readers. I also hit a bug / unintuitive behavior in the crate (maybe related to https://nest.pijul.com/pmeunier/zstd-seekable/discussions/1?). While trying to figure out how to solve that issue or contribute to it, I eventually looked into the seekable format more and realized that the format itself is dead simple
...so, I just published zstd-framed, which wraps zstd with a custom Rust implementation for the seekable format parts. It includes both sync and async reader and writer types.
Just wanted to chime in because I published the code under the Unlicense, so it should be pretty easy to adapt into this crate if anyone was interested (although probably with some tweaks to the interface-- I built something with lots of levers for my specific use case)
FYI: Support for the seekable format is now available on main with https://github.com/gyscos/zstd-rs/pull/310
Is it possible to use the seekable feature with zstd crate right now? I cannot see any feature flags that can be activated as of now.
Yes, it can be enabled with the seekable feature in zstd-safe, but not in zstd directly. It is also not really documented on docs.rs, local docs work though: cargo d -F std,seekable --open
I see, thanks for the answer!
Would it make sense to expose this feature in zstd in the features array in Cargo.toml? (similar to other features)
Also, is there a new release planned soon? 👀