amazonka-s3-streaming icon indicating copy to clipboard operation
amazonka-s3-streaming copied to clipboard

Reuse buffer for streamingUpload

Open axman6 opened this issue 2 years ago • 1 comments

In streamingUpload, a new buffer is allocated for every chunk in the call to finaliseS. A better idea might be to allocate one buffer and reuse it - at the moment there shouldn't be any possibility of concurrent access to the buffer between different requests.

Taking this one step further, instead of using the S abstraction, we could write directly into this shared buffer when each ByteString is received. This would mean needing to be very careful about ordering so we don't start writing the next chunk into the buffer before sending it, but I think that should be handled by yield passing the chunk to be sent before the conduit continues.

axman6 avatar Jan 06 '23 14:01 axman6

I would also be interested in concurrent uploads, even when drawing from a stream. Buffering N×chunksize blocks of memory doesn't seem that bad, but getting concurrent uploads and reused buffers right at the same time could be really hard.

endgame avatar Jan 06 '23 21:01 endgame