stream data

[nico202]

I'm trying to figure out how to stream data with Mux.jl

I want the body to be taken from IOBuffer until the buffer is close. Any suggestion?

[cmcaine]

You can stream data with websockets, but not with regular HTTP with Mux.

That's because serve(app) expects a function that goes HTTP.Request -> HTTP.Response and HTTP.jl expects the body of the response to be a Vector{UInt8}. Now, you can get a vector like that from an IOBuffer or by memory mapping some IO source, but I don't think you'll be able to make HTTP.jl wait until the buffer is closed.

If websockets don't suit your need, you could open an issue or PR with HTTP.jl, I don't think it's unreasonable that the response body could be Union{<:IO, Vector{UInt8}}.

[clarkevans]

@cmcaine This isn't an invalid ticket and WebSockets is not required to stream a response. There are reasons to want to stream a response. Here are the complications. First, the reason why HTTP.Response uses Vector{UInt8} is that it's pre-rendering the results in order to get the Content-Length header. This is a design choice for HTTP.Response. A more nuanced design choice for HTTP.Response would be to have the body be an object that one can print. HTTP could decide to have a buffer size of 16k or something, and use print(buffer, obj) to fill that buffer. If the printing terminates before the buffer is full, it could send the buffer. Alternatively, if the buffer fills before the printing has finished, HTTP could use chunked transfer encoding to send that chunk, and then send further chunks as needed, with a final chunk designating that the stream has complete.

The design of higher-level libraries should abstract away this design choice. I would accept for the content body any object that is printable (perhaps using show(io, mimetype, obj) if you know in advance what the mimetype is.

I made a ticket over at HTTP. https://github.com/JuliaWeb/HTTP.jl/issues/653

[cmcaine]

I think this is an invalid ticket for Mux because, as I said earlier and as you seem to agree, this is an issue for HTTP.jl to decide on.

HTTP.jl does offer a streaming interface, but you can't use it at the same time as the Request and Response objects.

[cmcaine]

HTTP.jl now allows you to provide the body as an IO value, which will then be written to the HTTP.Stream.

To use it from Mux, provide an IO value for the :body key in the response object.