httpx icon indicating copy to clipboard operation
httpx copied to clipboard

Add support for streaming multipart/form-data

Open fxn opened this issue 2 years ago • 0 comments

(After initial discussion in https://github.com/encode/httpx/discussions/2227.)

As far as I can see, there's no way to stream multipart/form-data uploads when the Content-Length is unknown, since HTTPX tries to get body size some way or another (last resort is to read it all in memory and count). Please, let me know if I overlooked something!

Use case would be to stream something whose size is unknown beforehand, like fetching and in parallel uploading a CSV export from a Postgres database, say.

HTTP/1.1 has chunked encoding for that. As you know, that streams the message itself, not the resource. This encoding is supported for "direct file uploads" in HTTPX, but not for multipart/form-data. This is strange, since chunked encoding is independent of Content-Type. Indeed, if both chunking and compression are in place, you first compress, and then you chunk the compressed bytes.

In HTTP/2 chunked encoding does not exist, byte size is per DATA frame. As you know, a message is allowed to have several. So, in this case the API would support uploading with multiple DATA frames, coming from a generator or iterator, for example.

fxn avatar May 30 '22 11:05 fxn