rust-s3 icon indicating copy to clipboard operation
rust-s3 copied to clipboard

put_object_stream reads stream to determine it's size

Open annmuor opened this issue 1 year ago • 1 comments

Describe the bug When using async put_object_stream I assume that the stream will be readed part by part as in async mode, but in reality it reads the whole stream to determine the size first. For large files it leads to OOM errors.

To Reproduce

let (mut rx, mut tx) = tokio::io::duplex(65535); // tokio duplex
tokio::spawn(async move { loop  { tx.write_all(&[1,3,4,5,6]).await; } });
Bucket::put_object_stream(b, &mut rx, "/123.txt").await;

Expected behavior Memory is not overflowing, data is written as you go.

Environment

  • Rust version: 1.69
  • lib version 0.33.0

Additional context

annmuor avatar Jun 07 '23 11:06 annmuor

This seems to be covered by #346

robinfriedli avatar Jun 11 '23 16:06 robinfriedli