WebMock with allow net connect mangles and unbuffers streamed responses

[SamSaffron]

require 'net/http'

# If you include this
# 1. you only get 1 chunk back instead of 5
# 2. the chunk you get is incorrectly compressed

require 'webmock'
WebMock.enable!
WebMock.allow_net_connect! # <== leads to error

uri = URI("https://www.discourse.org")
Net::HTTP.start(uri.host, uri.port, use_ssl: (uri.scheme == "https")) do |http|

  req = Net::HTTP::Get.new('/', {
    'Accept-Encoding' => 'gzip'
  })

  http.request(req) do |resp|

    resp.decode_content = true

    resp.read_body do |chunk|
      p chunk.length
    end
  end
end

Without webmock output is

16384
16384
16384
16384
5898

With webmock and allow_net_connect!

27369 

[bblimke]

@SamSaffron do I understand correctly that the problem only exists for gzipped responses?

[SamSaffron]

Nope, I think there are 2 problems here

1. It is functionally broken for gzip
2. Ruby splits stuff into 16k chunks and so should webmock ( it should probably be configurable)

On Tue, 30 Jan 2018 at 8:23 pm, Bartosz Blimke [email protected] wrote:

@SamSaffron https://github.com/samsaffron do I understand correctly that the problem only exists for gzipped responses?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bblimke/webmock/issues/742#issuecomment-361528938, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAUXWK4uadPgDD9kuteL_ZmecWWyx8gks5tPt9-gaJpZM4Rv6oU .

[babelfish]

I ran into this today, it also breaks streaming json responses. It gets combined into a single chunk which is an invalid json document.

[bblimke]

@babelfish could you please provide an example? why would a single chunk be invalid json?

[babelfish]

I was dealing with responses to the docker api, an endpoint that returns streaming JSON. Each chunk is a JSON object. If you concatenate them together you get an invalid JSON document, e.g.

{"some_field":"some value"}
{"some_field":"some value"}

[bblimke]

@babelfish thank you. That makes sense. Webmock's Response would have to hold a collection of chunks in order to handle that instead of a single response body string.

[alexeyr-ci]

We just ran into this issue. Happily, WebMock can be disabled in that test, but it took some time to find it was the source of the problem.