Deal with Body cloning / Allow phased sends from `HttpClient`

[Fishrock123]

As described in https://github.com/http-rs/surf/pull/195 (Redirects), but more severely in order to address https://github.com/http-rs/surf/issues/169 (Retries) some form of either body cloning and/or phased client sending is necessary.

The easier case is probably phased sending, required for Redirects. Phased sending would partially solve the issue for Retries, but only partially, although enough to implement a sort-of working Retry middleware.

The problem here is that in some cases we don't want the Body stream to be consumed, or at least not right when we send the request headers.

Ideal for Redirects:

• Send headers, await a partial response. If it's 3XX don't bother sending the body and go through the redirect first.
  • (I hope I understand http enough that the above is actually possible. If not, it is the same as Retry.)

Ideal for Retries:

• Body stream can be duplicated (Cloned), i.e. in the case of a file, a new FD to the same file, through the same transforms.
• Ideally it would be duplicate-able after the request is sent and fails, so that we don't have to allocate up front, but maybe that's just the price to pay for Retries.
• This is needed so that we can make a second identical request including body from an http_types::Request.

[yoshuawuyts]

The easier case is probably phased sending, required for Redirects.

Can you elaborate what you mean by this? I'm unfamiliar with the term.

[Fishrock123]

I don't know if there is a proper term, but:

Send headers, await a partial response.

It would be nice to not have to send the body if we don't have to, essentially.

[Fishrock123]

Another possible option for retires is to have that be behind a compiler flag that requires all body streams to be Clone... but honestly I'm not 100% sure if or how that would be possible for the streams to implement? Not sure.

[zkat]

I notice that node-fetch doesn't allow redirects for Readable stream bodies, which seems pretty sensible. I didn't know there was even such a thing as a partial response, though.

Just to add a point of interest: by default, curl will make the first POST request as a POST. If it's a redirect, it does the -following- one as a GET to see if it's another redirect, and then does a final POST on the final location. idk if that helps a lot, though.

[yoshuawuyts]

It would be nice to not have to send the body if we don't have to, essentially.

@Fishrock123 ahh, yes -- agree! @zkat ohh, I didn't know curl worked that way; that seems very reasonable!

[Fishrock123]

partial response

To be honest, I'm also not sure. I may be assuming some things about the http spec that aren't true. I am not sure that this is good idea, but a client could, for example, send part of the request, wait a bit to see if the server immediately responds with a 3XX, and if not, then send the body.

curl will make the first POST request as a POST. If it's a redirect, it does the -following- one as a GET to see if it's another redirect, and then does a final POST on the final location. idk if that helps a lot, though.

Any idea why it does this? Why not just POST?

I wonder how curl handles it's body/stream...

[zkat]

@Fishrock123 https://curl.haxx.se/libcurl/c/CURLOPT_FOLLOWLOCATION.html

I have no idea about either of your questions, but it definitely sounds like it might handle it?

https://curl.haxx.se/libcurl/c/CURLOPT_READFUNCTION.html is the API for setting the source "stream"