Kubectl Apply Lambda is being called multiple times leading to deployment failure

[mttwise]

Sometimes the apply lambda gets triggered multiple times causing the same manifest to be fed through twice. The issue appears to be intermittent, but has occurred more often recently. 50% of my deployments are seeing this issue on various stacks and have to be retried

The duplicate lambda run leads to an error like:

[ERROR] Exception: Error from server (AlreadyExists): error when creating "/tmp/manifest.yaml": namespaces "amazon-cloudwatch" already exists

Here are some sample logs...note that each lambda run was ~2 minutes apart and was triggered by the same CloudFormation stack deployment.

First Run of Lambda (Success)

Second Run of Lambda (Failure)

Stack Events: