aws-nuke
aws-nuke copied to clipboard
rebuy-de
Filter based on creation date
Our use case is to cleanup resources created by ci tests of CloudFormation resources.
The issue we have with the current filtering mechanisms is that we pretty much constantly have tests running, and without being able to exclude resources created recently nuke deletes resources that are part of in-progress ci builds, causing them to fail.
I realise that not all aws resources provide a creation date in the get/describe responses, for those that don't a possible workaround would be to store the timestamp for the 1st time nuke sees a resource and use that in absence of a definitive creation date. This obviously involves storing state somehow across invocations of nuke.
I think that storing the state of previous resources would increase the complexity of aws-nuke.
What we do in this case is to have a separate test account and configure all prerequisites for the ci-test. Then we configure aws-nuke to filter everything that we just configured. Running aws-nuke after the ci test would then clean every newly created resource.
Agree with @jaymccon about having an --older-than flag just like the https://github.com/gruntwork-io/cloud-nuke project has. That project is buggy as hell though, whereas this one is written quite well.
@svenwltr While this will certainly increase complexity somewhat, it could be done in an extensible and (fairly) decoupled way. I'd be happy to provide a proposal and implementation. If you're open to considering this, can you point me to some guidelines on what a proposal is expected to look like ?
As far as simply running aws-nuke after each test goes, this is not viable for our use-case as we have concurrent tests running at pretty much any point in time, would have to implement something to drain active tests from the account before running nuke. Even with that implemented we are then unable to keep resources related to failed runs for troubleshooting (currently we keep all failed resources for 2 days)
@jaymccon How would a filter based on the creation date help in your case? It wouldn't it break the concurrent tests, too?
@sc250024 I would rather put it in the config than as a flag.
Actually, the only issue I have with this feature is, that we would have to go through all resources and add this feature to each one. We currently have 276 resources. We already have this issue with general tag support, where we said we just add the tag support on demand. After pretty much one year we managed to add it to 31 resources. This is mainly because of a lack of time and necessity, but in my eyes it makes aws-nuke a bit inconsistent.
I noticed the dateOlderThan filter exists, but not all resources have the property needed or the name is different.
A global way to filter out (not delete) anything less than X days/hours old would be useful. Something where I don't have to specify a filter in my config for each different resource property name. Example: LaunchTime versus CreateTime versus ...
The goal being to delete anything older than say 30 days or 3 hours, etc.
Was there any progress made on this @rbeede? I am looking to do something similar, nuke resources older than account creation date(+2 days to safeguard initial resources configured by automation).
A number of the AWS APIs do not provide the necessary metadata for the resource creation date. If it is setup you have to query CloudTrail to find the history.
My workaround was to run code that documented the existence of a resource, tag when it was first observed, and then build a list of things to delete based on that observed date by myself.