zstd-rs icon indicating copy to clipboard operation
zstd-rs copied to clipboard

`stream::Encoder::recommended_input_size` requires specifying the writer generic

Open Firestar99 opened this issue 7 months ago • 1 comments

My use-case is to stream directly from an rkyv serializer through zstd into a file (10MiB to 5GiB compressed). I'm using stream::Encoder to do that, and noticed it has this nice method available:

/// Return a recommendation for the size of data to write at once.
pub fn recommended_input_size() -> usize { [...] }

I going to assume rkyv may only arrive with a few bytes each, so I thought it would be best to use a BufWriter before feeding it into zstd, like that:

BufWriter::with_capacity(
	zstd::stream::Encoder::recommended_input_size(),
	zstd::stream::Encoder::new(write, 0)?.auto_finish(),
)

EDIT: It lowered compression time from 10s to 2s for 300MiB.

However the example above does not compile, as recommended_input_size is within an impl block of Encoder, which requires specifying the writer type as a generic. Even though the method internally does not care for the generic at all. So instead you have to do something like this, which doesn't feel like how it's intended to be used, especially since the generic is the wrong type anyway.

BufWriter::with_capacity(
	zstd::stream::Encoder::<Vec<u8>>::recommended_input_size(),
	zstd::stream::Encoder::new(write, 0)?.auto_finish(),
)

Firestar99 avatar Jul 09 '24 13:07 Firestar99