perplexing interaction with IOBuffers

[StefanKarpinski]

I don't seem to be able to get a TranscodingStream wrapping an IOBuffer to flush any data out:

julia> using TranscodingStreams, CodecBzip2

julia> s = sprint() do outer
           inner = TranscodingStream(Bzip2Compressor(), outer)
           println(inner, "Hello, world.")
           close(inner)
       end
""

julia> s = sprint() do outer
           inner = TranscodingStream(Bzip2Compressor(), outer)
           println(inner, "Hello, world.")
           flush(inner)
       end
""

julia> s = sprint() do outer
           inner = TranscodingStream(Bzip2Compressor(), outer)
           println(inner, "Hello, world.")
           flush(inner)
           close(inner)
       end
""

julia> s = sprint() do outer
           inner = TranscodingStream(Bzip2Compressor(), outer, stop_on_end = true)
           println(inner, "Hello, world.")
           flush(inner)
       end
""

julia> s = sprint() do outer
           inner = TranscodingStream(Bzip2Compressor(), outer, stop_on_end = true)
           println(inner, "Hello, world.")
           close(inner)
       end
""

julia> s = sprint() do outer
           inner = TranscodingStream(Bzip2Compressor(), outer, stop_on_end = true)
           println(inner, "Hello, world.")
           flush(inner)
           close(inner)
       end
""

Works fine when writing to a file as long as there's a close call. Is this a bug or am I doing something wrong? I'm also a little unclear on if closing inner will cause outer to be closed or not in the end. If I'm reading the docs correctly, by default it will, but if you pass stop_on_end = true then it won't?

[StefanKarpinski]

It turns out you need to print an END_TOKEN to the stream in order to flush it. This is pretty non-obvious and it's unclear to me why calling flush shouldn't just work, but that's what I learned at some point so I figured I'd post it here in case someone else has the same issue.

[Pbellive]

I ran into this same issue as @StefanKarpinski. Just wanted to say thanks for posting about the need to print an END_TOKEN and mention that this is in the docs now: Explicitly finish transcoding by writing TOKEN_END.

However, I only found my way to that part of the docs via this issue. Having to write the end token was highly un-intuitive for me. It would be nice to make this a bit more explicit by having something like a finalize_stream function, rather than having to write a particular token to the stream in order to finalize it. I imagine this need to finalize a compressed stream is why flush can't just take care of this?

[nhz2]

With #178 if the stop_on_end = true keyword argument is set when constructing the transcoding stream, the underlying stream isn't closed when the transcoding stream is closed. Now your last two examples should work without needing to use TOKEN_END.