aws-cdk-rfcs

aws-cdk-rfcs copied to clipboard

Description

Assets which are uploaded to the CDK's S3 bucket and ECR repository are never deleted. This will incur costs for users in the long term. We should come up with a story on how those should be garbage collected safely.

Initially we should offer cdk gc which will track down unused assets (e.g. by tracing them back from deployed stacks) and offering users to delete them. We can offer an option to automatically run this after every deployment (either in CLI or through CI/CD). Later we can even offer a construct that you deploy to your environment and it can do that for you.

Proposed usage:

cdk gc [ENVIRONMENT...] [--list] [--type=s3|ecr]

Examples:

This command will find all orphaned S3 and ECR assets in a specific AWS environment and will delete them:

cdk gc aws://ACCOUNT/REGION

This command will garbage collect all assets in all environments that belong to the current CDK app (if cdk.json exists):

cdk gc

Just list orphaned assets:

cdk gc --list

Roles

Role	User
Proposed by	@eladb
Author(s)	@kaizen3031593
API Bar Raiser	@njlynch
Stakeholders	@rix0rrr @nija-at

See RFC Process for details

Workflow

• [x] Tracking issue created (label: status/proposed)
• [x] API bar raiser assigned (ping us at #aws-cdk-rfcs if needed)
• [ ] Kick off meeting
• [ ] RFC pull request submitted (label: status/review)
• [ ] Community reach out (via Slack and/or Twitter)
• [ ] API signed-off (label api-approved applied to pull request)
• [ ] Final comments period (label: status/final-comments-period)
• [ ] Approved and merged (label: status/approved)
• [ ] Execution plan submitted (label: status/planning)
• [ ] Plan approved and merged (label: status/implementing)
• [ ] Implementation complete (label: status/done)

---

Author is responsible to progress the RFC according to this checklist, and apply the relevant labels to this issue so that the RFC table in README gets updated.

How do we generate keys when uploading? Just random?

If it were deterministic, we could use lifecycle rules of object versions: https://aws.amazon.com/about-aws/whats-new/2014/05/20/amazon-s3-now-supports-lifecycle-rules-for-versioning/

That way objects are only considered for deletion if they are not 'current'.

Now that I think about it, versioning would only help if we were regularly re-uploading files. That's probably not the case?

Life cycle rules sounds like a good option for sure. The object keys are based on the hash of the contents of the asset, so to avoid uploading in case the content hasn't changed (code)

I'm not sure this works if there are two stacks, and only one is deployed with a new version of the asset.

In my mind, the other stack should still point to the old version of the asset (should not be automatically updated), but now the asset will be aged out after a while and the undeployed stack will magically break after a certain time.

---

Alternative idea (but much more involved and potentially not Free Tier): a Lambda that runs ever N hours which enumerates all stacks, detects the assets that are in use (from stack parameters or in some other way) and clears out the rest?

Now that I think about it, versioning would only help if we were regularly re-uploading files. That's probably not the case? I thought you meant expiration for non-current versions. The latest version still stays alive indefinitely.

But I think this runs afoul of reuse across stacks.

:+1: on garbage collecting lambda that runs every week or so, with ability to opt out and some cost warnings on docs and online

Seems risky. What if a deployment happen while crawling?

Yeah perhaps only collect old assets (month old) and we can salt the object key such that if a whole month had passed, it will be a new object?

Thinking out loud.... requires a design

I've got quite a few assets in my bucket now after a month or so of deploying from CD.

How do I determine which ones are in use, even manually? I can't seem to figure out the correlation between the names of the files in S3 and anything else I could use to determine what's being used. The lambdas don't point back at them in any way I can see.

I want to eventually write a script to do this safely for my use case, but absent a way of telling what's being used I'm stuck.

S3 now has lifecycle rules that can automatically delete objects a number of days after creation which might be a solution too.

How do I determine which ones are in use, even manually?

The ones in use are those referenced in active CloudFormation stacks deployed via CDK.

Those stack templates will include something like this:

    "GenFeedFunc959C5085": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Code": {
          "S3Bucket": {
            "Fn::Sub": "cdk-xxx-assets-${AWS::AccountId}-${AWS::Region}"
          },
          "S3Key": "5946c35f6797cf45370f262c1e5992bc54d8e7dd824e3d5aa32152a2d1e85e5d.zip"
        },

S3 now has lifecycle rules that can automatically delete objects a number of days after creation which might be a solution too.

Unfortunately, that won't help since old objects might still be in use, e.g. when a Lambda wasn't deployed in a while.

(It doesn't help that all assets are stored in the same "folder" either.)

Doesn't lambda copy the resources during deployment?

Doesn't lambda copy the resources during deployment?

The Lambda service will cache functions, but AFAIK there's no guarantee that they will be cached forever.

I'm fairly certain that Lambda only reads the assets during deployment and that they aren't needed afterwards. You can for example deploy to lambda without using S3 for smaller assets and those aren't stored in there.

Would love to read about the behavior in the docs somewhere. That 50 MB upload limit must exist for a reason. Haven't found anything so far though.

I can't find any concrete resources on this but I haven't found any docs mentioning that you cannot delete the object after deletion. Also, my lambdas doesn't have any permission to read my staging bucket nor does it mention that the object is from S3 so I doubt it's required to keep the object around.

I would suggest writing s3 objects as transient/YYYY/MM/DD/HASH.zip and having a lifecycle policy to remove transient/* files after 3 days. You'd get caching for builds done in the same day. This is similar to the salted hash suggested, but a lot more explicit, observable, and not subject to collisions. Also, the hash could stay the same day to day, so as to not trigger a spurious Lambda redeploy.

The main issue here is you need to pick your date / prefix only once, and stick with that, for the whole build process. You don't want to upload files at Tue 23:59 and a later process/function looks in Wednesday for the object. Perhaps just having an --transient-asset-prefix argument to cdk deploy would be enough?

Option 2 for S3 is to check the last-modifed date on objects, and just force a re-upload every N days. Then you could have a lifecycle policy to delete old objects (> N+1 days), and that shouldn't race with deploys. You'd need to be 100% sure the re-upload doesn't race with S3's lifecycle process, however. That's why I prefer the immutable objects in option 1, there is no chance of a race.

ECR is different because it's not transient, it is the long term backing store for Lambda. Just spitballing here, but if there was a "transient ECR repo" with a 7 day deletion policy, you could push new builds to that, and then during cloudformation deploy, those images would then be "copied/hardlinked" to a "runtime ECR repo" with lifetime managed by cloudformation, e.g. removed upon stack update/delete. Maybe the same thing could be accomplished if cloudformation could set ECR Tags that "pin" images in the transient repo that are in use, and tagged images are excluded from lifecycle rules. However, to avoid races, builds have to push something (e.g. at least some tiny change) to the transient ECR repo to refresh the image age (at least if the image is older than a day), so it won't be deleted right before cloudformation starts tracking / pinning it.

Word of caution: doing just a time based deletion on assets is a little risky. We have had this scenario play out:

• A stack is deployed and is healthy.
• The stack's assets are removed from S3.
• The stack is then updated at a later date (new assets in S3), but the update fails and triggers a rollback.
• The rollback fails because CFN looks for the old assets in the prior template.
• This puts your stack into the UPDATE_ROLLBACK_FAILED state. For us, we were able to get out of it by carefully skipping some rollback on resources. It's a rather scary state to have a stack in TBH.

So, unsure how you would even accomplish this, but ideally don't delete any S3 assets that are referenced in existing CFN templates.

So, unsure how you would even accomplish this, but ideally don't delete any S3 assets that are referenced in existing CFN templates.

Sorry, was reading quickly - sound like the RFC is going to try to do this, so yah!

If the only reason is to prevent CFN from "freezing" if role back is required, then just having a deletion policy of say 1 month should be okay given that there will be no scenario where our deployment frequency is less than a month.

So with the assumption that our deployment frequency is much quicker than a month, we can rest assured that no old assets are referenced within the current CFN so that if role back does occur, the assets will still be present within the bucket for it to complete.

We are having a discussion about this on the CDK slack here (link might not work in the near future): https://cdk-dev.slack.com/archives/C018XT6REKT/p1620291773488400

One solution a member (Julien Peron) is using, even though not ideal he said:

for this, i tried using tags on resources in cdk bucket in combination with life cycle policy. is not fully flawless, but may help you. It’s quite simple; I use one tag with 3 possible values: -current -previous -outdated At each deploy, I apply “current” value. If tag is already current, it becomes “previous”. If previous, it becomes “outdated”. Then the lifecycle policy deletes all “outdated” tagged objects

For the resource tagging, I tag all resources in the bucket at every deployment. I don’t have much resources and as the life cycle policy deletes the old ones, it’s quite fast. This method is just a one shot and is totally not bulletproof, but I believe there is something smart to do. In my case, I didn’t even add any checks before tagging. Which means, even if deploy fails, resources are tagged, which is not good. But yeah, just throwing the idea here to help. Here is a gist with the kind of script I’m using to tag: https://gist.github.com/julienperon/a048603f50ffe092a952d39672357618

Maybe his method if refined could work?

We are approaching 0.5TB of assets in the staging bucket. I can only imagine how much large companies have :(

See https://github.com/jogold/cloudstructs/blob/master/src/toolkit-cleaner/README.md for a working construct that does asset garbage collection.

In my projects, we use separate AWS accounts for each environment (testing, staging, production).

The use of assets somewhat leads you down the path of one branch per environment in your CI/CD pipeline.

A typical process might be:

• Commit application and infrastructure code to the main branch.
• This triggers a CI workflow in Github Actions or similar.
• The CI workflow builds the application code that's in the branch, and runs tests.
• The workflow then assumes a deployment role using the Github OIDC connector, or already has an IAM role in the case of using AWS CodeBuild etc.
• CDK builds and pushes Docker containers or Lambda entrypoints (as zip files) using CDK.
• The team has 3 AWS accounts, one for each branch:
  • main: testing AWS account
  • staging: staging AWS account
  • production: production AWS account
• To release code to the staging environment, the team merges or rebases from the previous branch:
  • main [merged to] → staging [merged into] → production
• This merge triggers the deployment to the next environment.

Problems with this approach

• At each deployment stage (main, staging, production), a full cycle of build and test is executed even though it was already built at the previous stage, wasting build minutes (and dev time since they're sometimes waiting for it).
• Duplicate copies of the Docker containers and Lambda zip files are built and deployed to each AWS environment as assets, wasting storage, and more build minutes / time.
• The Docker container for each version is not necessarily the same, since apt-get and other commands might produce different versions of dependencies.
• Discrepancies between application code package versions could creep in, e.g. Node.js applications that use a "greater than" syntax for versions ^1.0.0. We saw this with recent incidents like left-pad and, more recently, faker.js.
• All of the images are in the ecr-assets repo for each account, with human-unfriendly tags that make it hard to work out which image belongs to which application. I had to write a script to find out which image matches with which product, and clean out old ones.
  • https://github.com/a-h/cdk-ecr-asset-cleaner
• If the team has merge commits, the commit hash is different, so version numbers may also be different.
• It's hard to have a rollback mechanism, to move back to a previous version, e.g. you can't identify the previous version of the container.

Suggestions

The best thing about DockerImageAssets is the simplicity of it, and the low effort to get started. It's great to just have one command to run to build and deploy everything, but I don't see how to use it for multi-environment deploys without accepting that the built software might be different to the version you tested in another environment.

Traditional workflows involve building the application code separately from the infrastructure, which we want to avoid, because the infrastructure and software are often tightly coupled, e.g. a new feature needs to push to a newly created SQS queue, or use a new DynamoDB table.

I propose optimising for a workflow that's similar to this:

• The main branch builds the application code.
• The main branch then builds Dockerfiles and Lambda entrypoints.
  • For Docker assets, images are built.
  • For Lambda assets, zip files are built.
• The built assets are then pushed to a central repo:
  • For Docker assets, this might be Github Packages or similar (so that the build process doesn't need to have any access to AWS, or a separate build AWS account).
  • For Lambda assets, this might be an S3 bucket, or Github Packages containing the zips.
• Docker images are tagged with a version number, e.g.

export APP_VERSION=v0.0."`git rev-list --count HEAD`"-"`git rev-parse --short HEAD`

• Lambda zips would be placed in a directory of the stack and version number in S3.
• The CDK code is configured to use the specific Docker image tags and Lambda zip locations of the assets that were just pushed.
• The main branch CI pipeline packages the CDK code into a Docker container and tags that too.
• At this point, we have a CDK Docker image, and various assets:
  • project:v1.0.0-cdk
  • project:v1.0.0-asset-1-code
  • project:v1.0.0-asset-2-code
  • s3://name/stack/v1.0.0/function-name.zip
• The CDK Docker container would be configured to only deploy the matching application code containers and zips so that the application and infrastructure code don't diverge.
• Deployment can then be taken care of by a separate process which just runs the project:v1.0.0-cdk Docker image.
  • Only this CI process needs access to an AWS role.
  • This CI process would tag Docker images in the central repository with the account ID and account alias (name), so that you could see that the image was being used in the testing / staging environment etc.
  • To deploy the newly built project to another environment, e.g. staging, you'd point a deployment action in the staging environment to deploy the project:v1.0.0-cdk Docker image.

This workflow:

• Reduces the amount of asset builds, we only build the images and Lambda function zips once.
• Reduces the amount of asset storage, we only store it once.
• Guarantees that exactly the same assets are used in each environment.
• Enables the use of ECR and similar lifecycle rules, since any images tagged with account names (testing, staging etc.) can be protected.
• Makes rollback simpler, since you can redeploy past versions by deploying a different version of the CDK container.
• Makes it possible to identify which images are in use, and where by viewing the Docker image list, or listing the S3 buckets for Lambda functions.
• Gives friendly names to assets.

I think the issue with manually deleting artifacts out of the staging bucket is that there isn't an easy way to tell what's still in use. Part of the reason for this is that any single stack description .json may refer to lots of artifact .zip files including very old ones that haven't changed but are still part of the stack. This is an even worse problem if you have multiple stacks sharing a single staging directory.

I have three stacks sharing the same bootstrap/artifact directory. In retrospect this was a mistake but I didn't think to separate them when I started. Trying to work out a manual way to do this, the thought I have is to run get-template for each of your active stacks and note what resource files (.zip) are called out, for example:

  "stack1234lambdas3triggerC83C4999": {
   "Type": "AWS::Lambda::Function",
   "Properties": {
    "Code": {
     "S3Bucket": {
      "Fn::Sub": "cdk-hnb659fds-assets-${AWS::AccountId}-us-east-1"
     },
     "S3Key": "b202ff26ae16f03f3f28d7e48d3aeb9d47201b7084e2361973f7ccdb1d3b78ed.zip"
    },

I think if you collect all those S3Keys you'll have the list of what's still being called out, and you can remove the other artifact bundles without breaking CDK. There's still the problem of the old templates (I have about 400 .json template files currently), but I'm not sure whether or not you need to save those at all. It seems like a new one gets uploaded every time. My thought here is that those files are safe to delete just by date.

Hi - I was pointed to this issue / feature request by AWS Support. I see comments which pertain to cost concerns for the various assets being retained indefinitely, but wanted to raise another concern.

Our development team uses CDK for various services, and the indefinite retention of assets had gone sight unseen until discovered by me this morning. I operate in a security capacity, and recently implemented AWS Inspector continuous scanning on container images. Findings are configured to bubble up to Security Hub.

To my surprise, my Security Hub was flooded with findings for CVEs. I have thousands upon (tens of?) thousands of findings, specifically due to the lack of garbage collection of CDK ECR images. This is going to cause a constant game of whack-a-mole until:

1. I whitelist these images from the scanning (which I think is possible, and which maybe I should just go ahead and do?)
2. Garbage collection is implemented

Regardless, I wanted to bring another story for why garbage collection would be appreciated.

@createchange - that's how I initially noticed the problem too.

ECR's cleanup is bit silly, in that it can be configured to delete "old" containers, but doesn't have any understanding if they're in use or not, so I wrote a tool that only deletes containers that are not in use by Lambda and ECS Task definitions.

https://github.com/a-h/cdk-ecr-asset-cleaner

It doesn't tackle the buildup of S3 related assets, but it might be useful for you.

Has there been any movement on this? My Dev stack accumulated 144 undeleted images within a few days of people pushing test images. This involves quite significant costs which the DockerImageAsset construct does not make easy to resolve.

Thanks @createchange for pointing that out. I just encountered exactly the same issue.

I'd argue that this brings some security urgency to addressing this issue. The flood of spurious security alerts for images that potentially have not been deployed for years makes AWS Inspector basically unusable to protect our CDK assets in ECR.

Note there's another problem for the security inspector, and that is that it is very difficult to attribute what CDK component actually generated a given CDK assets image. This means that if a vulnerability is reported, we have to engage in detective work to find where it should be remediated.

We also experience this problem: hundreds of images in ECR of which only a handful are currently being used. Please provide some way to prevent this from spinning out of control!