S3Object.store $stdin

[tarnfeld]

trafficstars

Hey,

I've been trying to build a tool that allows me to put some data from STDIN into S3, and I have hit a brick wall with your S3 library. When I try to call AWS::S3::S3Object.store(path, $stdin, bucket) I get an exception thrown...

/Library/Ruby/Gems/1.8/gems/aws-s3-0.6.2/lib/aws/s3/connection.rb:41:in `request': undefined method `size' for #<IO:0x106464bd0> (NoMethodError)
    from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:543:in `start'
    from /Library/Ruby/Gems/1.8/gems/aws-s3-0.6.2/lib/aws/s3/connection.rb:52:in `request'
    from /Library/Ruby/Gems/1.8/gems/aws-s3-0.6.2/lib/aws/s3/base.rb:69:in `request'
    from /Library/Ruby/Gems/1.8/gems/aws-s3-0.6.2/lib/aws/s3/base.rb:88:in `put'
    from /Library/Ruby/Gems/1.8/gems/aws-s3-0.6.2/lib/aws/s3/object.rb:241:in `store'
    from ./s3pipe.rb:115

Everything works great if I call $stdin.read however the data i'm piping into my ruby program is upwards of 2GB, and i'm running this on low memory machines.

By the looks of things, your relying on a few methods that are only defined in the File class, and not handling raw IO objects. It would be aweseomeee if you could fix this bug :-)! I can't wait to open source this little ruby file i'm writing!

Thanks.

[kindkid]

@tarnfeld, as a work-around, check out the aws-sdk gem.

http://docs.amazonwebservices.com/AWSRubySDK/latest/AWS/S3/S3Object.html#multipart_upload-instance_method

Unfortunately, the aws-sdk gem currently doesn't support unbuffered download from s3. I ended up having to use both aws-s3 and aws-sdk gems when I made my s3stream gem. The namespace collision was not fun.

https://github.com/kindkid/s3stream