aws-nuke
aws-nuke copied to clipboard
rebuy-de
Parallelise API requests
Currently aws-nuke is rather slow, because it does every request sequentially. This means the process is sleeping the most of the time. Also the AWS API can probably handle a lot more requests (unfortunately I did not find any reliable documentation about this).
By parallelising the API request we can speed up aws-nuke a lot.
based on the experience from terraform there is a soft limit of 10 API calls in parallel , if you are looking at a start value
@svenwltr we need to be able to control this PER resource. Some allow 1 request/s tops I have found
I tried a very naive approach and just paralyzed all remove queries, regardless of the resource type. The problem is not just the rate limiting (ThrottlingException: Rate exceeded), but also that some resource need to generate a token, which expires as soon as a second one is requested (WAFStaleDataException: The input token is no longer current.)
I am not sure how to continue here. Currently, I do not see an easy way to independently parallize per service with the current code base.
Parallel is really needed particularly going to overseas regions. What about adding switches on resources which we know are not “parallel safe”. We can put it into stuff like WAF.
That’s the only one I’m aware of in my batch which operates like that, so it should be easy to default it to true. Wouldn’t hurt to add a config item which also disables it for the users if they are having an issue
On Fri, Jul 27, 2018 at 6:25 AM Björn Häuser [email protected] wrote:
Well, than no parallel?
— You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/rebuy-de/aws-nuke/issues/151#issuecomment-408378716, or mute the thread https://github.com/notifications/unsubscribe-auth/AAUQRZSPyUBIS5FOUQebIUiW27CBoeWxks5uKuq0gaJpZM4SzRJr .
Hm, I understand. Which APIs come to your mind?
I would only see:
- S3
- EC2
It’s really all of them. All of the newer APIs are highly scoped and interdependent. Say you are in the US cleaning the up a Mumbai region setup from your offshore team. It’s not going to end well. Most people I’ve talked to are using this for sandbox accounts or pre-deletion which means you hit every api in every region to ensure you got everything
On Mon, Jul 30, 2018 at 3:02 AM Björn Häuser [email protected] wrote:
Hm, I understand. Which APIs come to your mind?
I would only see:
- S3
- EC2
— You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/rebuy-de/aws-nuke/issues/151#issuecomment-408766391, or mute the thread https://github.com/notifications/unsubscribe-auth/AAUQRWHrFFCzWwvw1DVFbuAKCegCqwd1ks5uLq9sgaJpZM4SzRJr .
Not sure what the status of parallelization is these days. But I get a bunch of Rate Exceeded errors when nuking an account.
