Add support for uploading files to S3

[Joseph-R]

trafficstars

Hello,

We stand up dynamic clusters that have config files we want to place on S3, so that they can be read and registered by another service.

I see the aws_s3_file resource for pulling files from S3. But what about pushing files from a node to a bucket?

Regards, Joe Reid

[Joseph-R]

"Why not just upload the configs directly?"

The catch is that these config files are dynamic and change often as nodes enter and leave the cluster, we change configs, etc. We want to keep them in sync with S3, so that the API client that reads those files always has accurate information.

Since most of the changes we make are associated with Chef in one way or another, it makes sense to just tack the compiled config upload onto our run lists with a :delayed flag.

[jason-riddle]

I'm interested in picking this up, but I'm wondering on the approach? I was thinking in providers/s3_file.rb just adding the following.

action :post do
  up_s3_file(:post)
end

...

def up_s3_file(resource_action)
  md5s_match = false

  s3url = s3_obj.presigned_url(:post, expires_in: 300).gsub(%r{https://([\w\.\-]*)\.\{1\}s3.amazonaws.com:443}, 'https://s3.amazonaws.com:443/\1') # Fix for ssl cert issue
  Chef::Log.debug("Using S3 URL #{s3url}")

  http_request new_resource.name do
    message new_resource.path
    url s3url
    action resource_action
  end
end

It's not the cleanest approach so I'm open to suggestions.

[ton31337]

What's the progress integrating file upload?

[majormoses]

It looks reasonable to me from a quick look (though I need to take another look at that regex later), I can think of use cases to expose headers and sensitive as the signed request includes authentication and would be sent to a logging server.

[DeWaRs1206]

Any progress here?

[jgh9]

This is a great idea, and something I would love to see implemented. Currently, it is difficult to maintain code that is uploaded to S3, without doing other implementations, and track those changes in a meaningful way and can be shared through code collaboration. This bug would address that issue specifically, however I would urge support of http, as well, as that is supported through S3.

[majormoses]

This is a great idea, and something I would love to see implemented. Currently, it is difficult to maintain code that is uploaded to S3, without doing other implementations, and track those changes in a meaningful way and can be shared through code collaboration. This bug would address that issue specifically, however I would urge support of http, as well, as that is supported through S3.

Probably unpopular (security) opinion: what's the cost of https to an s3 bucket in AWS? I am strongly opposed to supporting http on s3 without a compelling reason. This has clear security risks with no benefit than I can see.

[jgh9]

This is a great idea, and something I would love to see implemented. Currently, it is difficult to maintain code that is uploaded to S3, without doing other implementations, and track those changes in a meaningful way and can be shared through code collaboration. This bug would address that issue specifically, however I would urge support of http, as well, as that is supported through S3.

Probably unpopular (security) opinion: what's the cost of https to an s3 bucket in AWS? I am strongly opposed to supporting http on s3 without a compelling reason. This has clear security risks with no benefit than I can see.

While unpopular, it is supported through AWS. This doesn't mean that objects aren't encrypted, themselves, just that transport layer is not. The cost is to businesses to support option of SSL-only under shared security model of AWS. Supporting a method, until it is unsupported upstream, is a perfectly acceptable support model. While I agree it is unpopular, there is little reason not to support it while it is allowed through platform, itself.

[Stromweld]

Just to help set expectations with sous-chefs having so few contributors and other day jobs Issues/requests like this aren't likely to be picked up unless it affects someone directly. If anyone would like to work on this we'd be happy to accept PR reviews to implement this.

I do agree this would be a nice feature to add potentially as a helper library that could be used in other resources.