amazon-eks-ami icon indicating copy to clipboard operation
amazon-eks-ami copied to clipboard

Published 20 hours ago • verified publisher icon awslabs

Iptables rules in kubelet.service

Open SMR39 opened this issue 5 years ago • 7 comments

Hello,

I was trying to understand the need for having this iptables rule here. https://github.com/awslabs/amazon-eks-ami/blob/master/files/kubelet.service#L8

If this was introduced to fix docker trying to change rule from ACCEPT to DROP then I think this was indeed fixed in https://github.com/kubernetes/kubernetes/issues/59656 and that line can be removed to avoid further confusions.

SMR39 avatar Sep 11 '19 13:09 SMR39

I do think we should verify this. The change seems to be

https://github.com/kubernetes/kubernetes/pull/62007/files#diff-97649d13432b01b4e7669d728a2e2e82R1324-R1329

mogren avatar Apr 27 '20 22:04 mogren

We are noticing that kubelet runs pretty close to when iptables restore runs. Due to the close proximity, we notice there are nodes with FORWARD DROP that came back from the restore (we think) since it may happen after kubelet's FORWARD ACCEPT. Given that kubelet and ipstable-restore both are only depending on docker service to finish https://github.com/awslabs/amazon-eks-ami/blob/master/files/kubelet.service#L4 https://github.com/awslabs/amazon-eks-ami/blob/master/files/iptables-restore.service#L5

Not sure if anyone is running into this or if this is the cause of the issue

GnatorX avatar Aug 22 '20 04:08 GnatorX

An update on the issue I raise. It seems like, for us, an iptable save was performed prior to startup of the node which meant we saved FORWARD DROP. On startup, since kubelet and iptable restore happens around the same time, we run into a race condition where FORWARD DROP may be applied over FORWARD ACCEPT if iptable restore after kubelet start.

GnatorX avatar Aug 26 '20 17:08 GnatorX

I have another question, in https://github.com/awslabs/amazon-eks-ami/blob/v20210830/scripts/install-worker.sh#L99, I'm curious why only a copy there without starting the service? How the service been started?

yongzhang avatar Sep 15 '21 03:09 yongzhang

I have another question, in https://github.com/awslabs/amazon-eks-ami/blob/v20210830/scripts/install-worker.sh#L99, I'm curious why only a copy there without starting the service? How the service been started?

Sorry, found it in bootstrap.sh.

yongzhang avatar Sep 15 '21 03:09 yongzhang

Is it a really good idea to have a move here https://github.com/awslabs/amazon-eks-ami/blob/v20210830/scripts/install-worker.sh#L99 (as opposed to having a copy)?

It cost me a day to figure out why re-running bootstrap.sh was failing to start my cluster :/

hovo972 avatar Dec 17 '21 21:12 hovo972

I don't know why this was added in #90, I think the reviewers overlooked it. It's not clear to me why this would be necessary, as components like kube-proxy that rely on iptables rules should be managing those themselves.

@nithu0115 was the author, who might be able to provide more info. Otherwise, I'll try removing it when I have some time.

cartermckinnon avatar Dec 08 '22 02:12 cartermckinnon