Add transparent Content-Encoding: gzip/deflate support

[seriyps]

So, should add Accept-Encoding: gzip,deflate to request, then check for Content-Encoding: gzip or Content-Encoding: deflate and decompress body (after de-chunking). Decompression may be done in single pass (zlib:gunzip/1 for gzip and zlib:unzip/1 for deflate) in case of synchronous requests and using streaming (zlib:open/0, zlib:inflateInit/1, zlib:inflate/2 ..., zlib:close/1) in case of asynchronous. Single-pass example can be found here https://github.com/seriyps/xhttpc/blob/master/src/middlewares/compression_middleware.erl

Some option should be added to enable this feature.

[benoitc]

will add it in next release. Thanks for the suggestion.

[zweizeichen]

+1

[seriyps]

Also, bear in mind new Erlang 18.0 API to protect from gzip bombs http://www.erlang.org/documentation/doc-7.0-rc1/erts-7.0/doc/html/zlib.html#inflateChunk-2

[benoitc]

feature is coming. I am in the middle of adding some big improvments to hackney. I will probably include it right after. Thanks for the hint.

[tuscland]

+1 :)

[zyro]

+1 Would be great to see this handled transparently for both requests and responses.

[jedhu0]

+1 :)

[Anonyfox]

+1 on this :)

[kostya]

+1

[edgurgel]

Can I help with this? I would be glad to follow some guidance and help with this feature.

[benoitc]

@edgurgel thanks! It should probably be added to the function receiving the body, if you wrap them and keep the state it should be OK. Anyway betetr to wait over the we, with the new recv and recv_multipart functions :)

[tlvenn]

Any update on this feature ?

[loicvigneron]

+1 :)

[stevegraham]

What's the status of this issue?

[benoitc]

No work have been done on my side yet. But any help is appreciated. Anyway I expect that this feature will land after the current rework of the internals.

[sunboshan]

+1. Just encountered this issue that some server send gzip content and Hackey returns bitstring in body. Finally figure the root issue with lots of people's help. Though it would be really good if Hackey can transparently unzip the body based on Content-Encoding. :)

[jimdigriz]

I have a shim wrapper for this that works for default/gzip and even when calling stream_body/1 that I can put up here if or anyone wants to see the moving parts that are involved? It is not a hackney fork though.

[edevil]

Have the internals been reworked?

[cgorshing]

I'm working on a ueberauth strategy for StackOverflow and I would love to have this feature completed so I don't have to wire it into OAuth2's handling of the response.

I have some time available to actively help develop, test, or otherwise pitch in. How might I be able to lend a hand? Is a branch started for this that I can begin looking at?

[jimdigriz]

Curious what the benefit of gzip/deflate is for Oauth2 flows?

On 29 October 2017 11:19:57 p.m. Chad Gorshing [email protected] wrote:

I'm working on a ueberauth strategy for StackOverflow and I would love to have this feature completed so I don't have to wire it into OAuth2's handling of the response.

I have some time available to actively help develop, test, or otherwise pitch in. How might I be able to lend a hand? Is a branch started for this that I can begin looking at?

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/benoitc/hackney/issues/155#issuecomment-340311482

[cgorshing]

Responses from StackOverflow are gzipped. Ueberauth hands off the request to the OAuth2 library, that just calls Poison for the deserialization of the body. The OAuth2 library does not pass the headers to Poison, just the body.

I'm getting around this by creating my own serializer to gunzip the body before handing off to Poison (https://github.com/scrogson/oauth2/blob/master/lib/oauth2/serializer.ex#L27-L39). But in doing this, I still don't have access to the headers so it's blind faith right now.

Hit me up on Elixir's slack if you are more interested, I feel like this is a tad off-topic for this issue.

[jimdigriz]

Surely only when you include a accept-encoding: gzip, deflate header though?

[fenollp]

@jimdigriz Can you share your shim with us?

[jimdigriz]

Okay...it is ugly as sin though. Attached is code you will just have to adapt to fit into your setup.

zlib-shim.txt - lets call this the http module

So instead really what you want is:

• hackney:get -> http:get (you cannot use the get body option here)
• hackney:body -> http:body
• hackney:stream_body -> http:stream_body
• http:stream_body_drain is used for when you need to drain and discard a request; though probably show not exist as you could just call http:body...

[jimdigriz]

Anyone tried https://github.com/benoitc/hackney/pull/456, it has been in production for my use case for two weeks?

[tsloughter]

Was curious if this was supposed and found this issue. Is it still planned for 2.0.0?

[benoitc]

i am lagging in release, but yes that will be part of the next release. There is a RC that will land next week including it :)

[KirillDavydov]

+1