perfom disable src-dst check on single node

[murali-reddy]

Perfom AWS EC2 disable src-dst check on single node when we recieve node updates from API server. In the current implementation is attempts to disable for all the nodes, in case when there is update from API server on node add/update event.

[zegl]

This bug (or lack of feature) caused an outage for me recently.

A rolling restart of kube-router was performed to get a configuration change out. When the new pods started all of them tried to disable the src-dst-check for all nodes at the same time. We quickly reached the AWS rate-limit of performing this action, as an isolated event that is not too bad, but this time new nodes where also starting up soon after, and they where not able to disable the src-dst check. This caused many nodes to enter a state where they could not use the network.

[recombinationgroup]

We were also just bitten by this in our kops-provisioned cluster on AWS. For anyone experiencing the same issue: we had an additional policy attached to our worker nodes, which granted ec2:ModifyInstanceAttributes. Removing this specific permission is a valid workaround, as explained by @murali-reddy here: https://github.com/cloudnativelabs/kube-router/issues/536#issuecomment-421402764. This fixed the issue for us.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stale for 5 days with no activity.