Implement streaming response compression

[noteflakes]

def respond_gzip_from_io(io, opts)
  r, w = IO.pipe
  spin do
    writer = Zlib::GzipWriter.new(w)
    io.feed_loop(writer, :<<)
    w.close
  end
  serve_from_io(r, opts)
end

For deflate, we can use the Deflate#deflate:

def respond_deflate_from_io(io, opts)
  deflater = Zlib::Deflate.new
  io.read_loop do |data|
    deflated = deflater.deflate(data)
    adapter.send_chunk(deflated)
  end
  deflated = deflater.flush
  adapter.send_chunk(deflated, done: true)
end

[noteflakes]

#respond_deflate_from_io could be rewritten using #feed_loop:

def respond_deflate_from_io(io, opts)
  deflater = Zlib::Deflate.new
  io.feed_loop(deflater, :deflate) { |deflated| adapter.send_chunk(deflated) }
  deflated = deflater.flush
  adapter.send_chunk(deflated, done: true)
end

[noteflakes]

An interesting experiment would be to add an example Tipi app for benchmarking different ways of sending a zipped response:

• ~~Three~~four file sizes: 1024K, 128K, 1M, 16M
• The following methods for sending zipped response:
  • Existing in Qeweney (read file, zip it, send it)
  • Use ZipWriter, read chunks to string, emit to writer, read from writer, write to response
  • Use ZipWriter with a pipe

[noteflakes]

Adding some custom C-level usage of zlib is also an interesting direction. The API looks quite simple:

https://zlib.net/zlib_how.html.

Performance wise there's a good chance that if we find a way to integrate this kind of feature into the Polyphony runtime, we could take on-the-fly compression/decompression to the next level: minimal overhead in terms of pressure on the GC, and we don't need to go back and forth between Ruby method calls.

So the kind of API will have will maybe look like this:

# src and dest are both IO instances (or sockets or SSL sockets)
IO.deflate(src, dest)
IO.inflate(src, dest)
IO.gzip(src, dest)
IO.gunzip(src, dest)

Then we can implement a gzipped reply as something like:

def respond_gzip_from_io(io, opts)
  r, w = IO.pipe
  spin do
    IO.gzip(io, w)
    w.close
  end
  serve_from_io(r, opts)
end