Allow access to chunked transfer trailers

[daxhuiberts]

I am talking to a webserver which uses chunked transfer encoding with an additional trailer, similar to what is explained here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Trailer

But I'm getting a "Error while decoding chunks" when requesting it with ureq. It's a bit similar to https://github.com/algesten/ureq/issues/325, but in this case the response is correct. Before the last '\r\n' the trailing headers are added to the response stream, which are then ended with a double set of '\r\n'.

I'm not necessarily interested in the trailing headers, but I am interested in properly reading the response without getting an error.

From what I've read in https://github.com/algesten/ureq/pull/454 and https://github.com/frewsxcv/rust-chunked-transfer/pull/11 the double '\r\n' handling is done in the rust-chunked-transfer crate, but proper trailing handling is not the concern of the rust-chunked-transfer crate. That means that the rust-chunked-transfer crate should not or allow not to handle the double '\r\n' handling so ureq can process the trailing headers.

I'm not sure what the best approach is here or whether this is something you'd like to handle in ureq. I'd like to pick this up if it's clear what the approach would be.

[algesten]

At this point I'm open to forking the upstream project into a local mod or just write our own take.

The one danger is request smuggling, care must be taken to ensure that's not possible.

[jsha]

I'm in favor of accepting chunked responses with trailers, and maybe eventually providing access to them, though the interface would need some thought.

I happen to have a branch where I already forked rust-chunked-transfer into a local mod, in pursuit of #559. I'll put that up for review now, independent of the #559 changes (which still need some work).

[algesten]

ureq 3.x is accepting trailers, but they are skipped. This shouldn't be too hard to fix.