AWS::S3::S3Object.store doesn't support transfer encoding chunked

[taf2]

trafficstars

When using AWS::S3::S3Object.store with a generic IO object - say a IO.pipe, store should use transfer encoding chunked since it can't compute the size of the read end of the pipe. This would allow people to stream uploads. The example use case is you are downloading files from one machine via net/http net/scp or net/ftp... instead of downloading the full document and writing it to disk it would be nice to stream those files down in chunks and upload the chunks directly to S3 rather than download them completely and than upload them.

[marcel]

Your reasoning, at first glance, seems sound. Provide a tested patch and I'll work with you to get it merged in. Thanks.

[taf2]

I figured out even though it would be nice to stream this way, S3 doesn't allow Transfer-Encoding:chunked when uploading... Still if it was possible to than provide a predetermined content-length and than using a pipe and 2 threads it would be possible to stream files from say ftp to s3... I'll see about a patch... for now I've hacked up a solution using curb but this would work with just about any net http library... https://gist.github.com/841401