TranscodingStreams.jl icon indicating copy to clipboard operation
TranscodingStreams.jl copied to clipboard

Allow extra keyword arguments in the Transcoding protocol

Open nhz2 opened this issue 1 year ago • 2 comments

This PR adds the ability to extend the Transcoding protocol in a non-breaking way.

This is desirable to support non-streaming codecs better. Ref: https://github.com/JuliaIO/CodecZstd.jl/pull/46

To do this I added two new optional functions to the protocol: process2 and startproc2. These are equivalent to their previous versions, except they can accept any keyword arguments.

These are optional to maintain compatibility with existing codecs.

~~This PR also sets the mode to :transcode instead of :write when calling startproc in unsafe_transcode! This is to allow non-streaming codecs to error if they are used in a streaming context. Ref: https://github.com/JuliaIO/CodecZstd.jl/pull/46~~

Edit: I'm removing the :transcode mode change because this can be a different PR.

nhz2 avatar May 15 '24 03:05 nhz2

Can you give me an example of how this would work?

mkitti avatar May 16 '24 22:05 mkitti

I'm working on a CodecCBlosc1 to see how new keyword arguments could be used.

The goal is to allow a compressor Codec to avoid buffering input if there is no more input in a frame and enough output space is available.

One thing I found is that any input passed to process needs to first get passed to minoutsize, so I'll need a minoutsize2 as well.

nhz2 avatar May 18 '24 23:05 nhz2