Is there a way to not buffer?

[ralt]

I am streaming from one URL hosting big file to s3, and would like to avoid buffering locally, either in files or in-memory. Given that there is no obvious API for this, I tried to trick it with a fifo, but I assume this thing is messing things up.

Is there a way to do this? It feels like I would have to write some code there, but I'm not sure if maybe e.g. s3 API has some limits with regards to big files uploads.

[ralt]

Ok, the funny part is that this part is what is actually breaking the fifo:

  (ensure-amz-header request "content-sha256"
                     (payload-sha256 request))

Because it's reading it, then put-object is just blocked because the fifo was already depleted. Looks like I need to make the request myself without having this header.

[ghard]

I was just researching this myself - apparently the S3 API has what is called multi-part upload, which doesn't remove the need for buffering altogether, but lets one break the upload into more manageable chunks which are then assembled into a single object. I don't see support for this API in present ZS3 but am exploring implementing it on top of it, as I may well have need for it in not too distant future.