aws-extensions-for-dotnet-cli icon indicating copy to clipboard operation
aws-extensions-for-dotnet-cli copied to clipboard

Support versioned S3 artifacts for lambda package-ci code uploads

Open hauntingEcho opened this issue 3 years ago • 5 comments

Allowing the use of versioned S3 buckets for code artifacts could greatly simplify management of old versions & deduplication of new versions.

Describe the Feature

When you repackage the same code artifact, it could be uploaded as a new version of the same S3 object

Is your Feature Request related to a problem?

Managing version history across many thousand timestamped items is much more complex than using a single versioned S3 object.

Proposed Solution

  • If versioning is enabled for the target bucket, don't add a timestamp to the target location in UploadToS3Async: https://github.com/aws/aws-extensions-for-dotnet-cli/blob/d32048587fcbac721e183dcb260d3f6868cabfc6/src/Amazon.Common.DotNetCli.Tools/Utilities.cs#L556
  • Don't add a timestamp to the generated zip file name here: https://github.com/aws/aws-extensions-for-dotnet-cli/blob/6b3c0eaf04bb62995ca1c5dd71b0db7e564e8b74/src/Amazon.Lambda.Tools/TemplateProcessor/TemplateProcessorManager.cs#L260

Additional Context

We currently have many thousand objects in our S3 buckets from our CI system. Managing versioned artifacts is a lot simpler, particularly where ETags are able to be file hashes. Making these changes may also make it a lot simpler to only upload changed artifacts, per #83


This is a :rocket: Feature Request

hauntingEcho avatar Oct 01 '20 16:10 hauntingEcho

correctly updating the template's code URI would probably require changes in the SDK as well, to return the result of the underlying put operation: https://github.com/aws/aws-sdk-net/blob/7c88eeceb52869892adb1fe50c3f06c8f1a6b126/sdk/src/Services/S3/Custom/Transfer/Internal/_async/SimpleUploadCommand.async.cs#L42

hauntingEcho avatar Oct 01 '20 17:10 hauntingEcho

We have noticed this issue has not recieved attention in 1 year. We will close this issue for now. If you think this is in error, please feel free to comment and reopen the issue.

github-actions[bot] avatar Oct 02 '21 00:10 github-actions[bot]

bump

hauntingEcho avatar Oct 03 '21 03:10 hauntingEcho

Appears to be a useful feature request. Needs discussion with the team.

ashishdhingra avatar Jan 20 '22 20:01 ashishdhingra

It is a valid feature request since Lambda supports specifying the location in S3 with a version id. When checking to see if the bucket has versioning enable we will need to be defensive for lack of IAM permissions to describe the bucket. If we get a permissions error then fallback to the current behavior.

normj avatar May 06 '23 00:05 normj