Security group already exists error

[JulianWgs]

When the initialization of the cluster fails for some reason, but the "dask ecs" security group has already been created on following attempts the initialization fails due to the security group already existing. To get around this it is possible look up the id of the security group and give as an argument in cluster initialization.

I think it would be better a) to look for an existing security group or b) to delete the security group when failing to initialize the cluster.

I think a) is more robust :)

Greetings

[martindurant]

Seems to me that this should happen at the cleanup stage of a cluster (whether the cluster was successfully created or not) and giving a nice meaningful error if that group already exists when trying to make a new cluster. Otherwise, you might end up breaking some existing cluster.

[JulianWgs]

That's true. I was thinking about an abrupt initialization, where no clean up code is run. May be add a clean up function which can be run whenever.

[martindurant]

May be add a clean up function which can be run whenever.

That's a good idea, but I still wouldn't run it automatically. Either the exception says "could not create security group because it already exists - you can clean up by ..." or provide an optional kwarg which deletes before creation ("clobber=False"?).

[martindurant]

Did you plan to submit something for this, @JulianWgs ?

[JulianWgs]

Sorry, I'm not planning to submit something, because I switched to AWS EKS.

[martindurant]

OK, well this thread can stay here for when someone else wants to attempt it - thanks for posting. My experience with EKS was less than satisfactory, so good luck!