Add --preserve-encoding option to keep response body encoding

[trueb2]

Changes

Add a --download-encoding flag to choose an Accept-Encoding header other than "identity" during a download request. During the download, the "decompress" function that reads the response and writes it to the output will passthrough data in whatever Content-Encoding was received if --download-encoding was specified.

Motivation

When downloading a large file, I want to be able to transfer it compressed and leave it compressed. For example, stream download a 500MiB CSV file in the gzip encoding, saving it to a data.csv.gz file.

jwtrueb@jmbp xh % cargo run --release -- get http://localhost:3000/download/csv/asdf -d --download-encoding gzip
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Type: application/csv
Date: Tue, 29 Nov 2022 23:17:12 GMT
Transfer-Encoding: chunked
Vary: accept-encoding

Downloading to "asdf"
Done. 147B in 0.00177s (80.92KiB/s)
jwtrueb@jmbp xh % gzcat asdf
ts,user_id,device_id,source,pressure,temperature,humidity
2022-11-28T20:13:12.704030+00:00,2,5,1,99999.0,25.3,16.4
2022-11-28T20:14:02.446438+00:00,2,5,1,99699.0,25.1,16.2

[blyxxyz]

One reason to use Accept-Encoding: identity is to be able to resume downloads (with --continue). We'd have to figure out how these features interact. Maybe that should be an error message.

This option seems too specific. Maybe we should allow overriding the encoding via a normal header argument and add a --preserve-encoding/--no-decode/--verbatim-body/whatever option? Then you'd be able to run xh -d --preserve-encoding :3000/download/csv/asdf accept-encoding:gzip.

We have to coordinate this with HTTPie, that's where we got the current behavior. Are you interested in making an issue?

[trueb2]

One reason to use Accept-Encoding: identity is to be able to resume downloads (with --continue). We'd have to figure out how these features interact. Maybe that should be an error message.

This option seems too specific. Maybe we should allow overriding the encoding via a normal header argument and add a --preserve-encoding/--no-decode/--verbatim-body/whatever option? Then you'd be able to run xh -d --preserve-encoding :3000/download/csv/asdf accept-encoding:gzip.

We have to coordinate this with HTTPie, that's where we got the current behavior. Are you interested in making an issue?

Sure whatever works. I couldn't figure out how to get HTTPie to do what I wanted as easily as xh. Looks like it wouldn't be too bad to switch this to a --preseve-encoding and a manual header. That still seems to be in the spirit of the CLI.

I don't really care to use --continue for downloads, so I did not consider how that interacts with anything other than identity. I think an error message makes sense if both --continue and --preserve-encoding are used.

I could just implement that instead of making an issue. Is it a problem to have features in xh that aren't in HTTPie?

[ducaale]

Is it a problem to have features in xh that aren't in HTTPie?

We are open to adding features not supported in HTTPie. But we don't want to end up in a situation where the same feature is implemented differently in xh and HTTPie. Many xh users are coming from HTTPie, and some of them might even rely on HTTPie docs while using xh.

Let's either open a new issue or maybe leave a comment on https://github.com/httpie/httpie/issues/178, and then we can continue with --preserve-encoding/--no-decode/--verbatim-body/something else.

[trueb2]

@ducaale That is quite the old issue you dug up! I see it is related to https://github.com/ducaale/xh/pull/130

I think we could comment on the xh approach in the httpie issue, then implement an approach. It seems like this isn't a very hot feature request, and I don't want it to end up as another issue that goes 9 years with an implementation as easy as this. What is your preferred implementation/flag?

[ducaale]

We can go with the proposed implementation in https://github.com/ducaale/xh/pull/294#issuecomment-1331882126 and use --preserve-encoding (cURL uses --raw but that option is already used in HTTPie/xh).

Note that we currently don't allow the user to set accept-encoding in download mode (see https://github.com/ducaale/xh/pull/141). We can roll back that behavior and error when --continue is used with a custom accept-encoding.