kubeadm Move Control Planes taint to kubelet config instead of markcontrolplane phase

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

What happened?

Due we apply the control plane taint after the control plane comes up, on a multi-control plane case we can have pod scheduled to this control plane.

What you expected to happen?

Use the kubelet --register-with-taints config instead of handler it on a separate phase.

How to reproduce it (as minimally and precisely as possible)?

Kubeadm init
Create a daemonset
Join another control plane

Anything else we need to know?

For now, I just use this config but would be nice if kubeadm can handler it.

apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
nodeRegistration:
  # Remove the default control-plane taint so we taint it manually with KUBELET_EXTRA_ARGS
  taints: []
```

Jun 19 '19 14:06 yagonobre

i guess this can be a real problem. thanks for the report @yagonobre

does the kubelet allow self-taints with node roles such as node-role.kubernetes.io/master=:NoSchedule? (it certainly does not for labels https://github.com/kubernetes/kubernetes/issues/45361)

Jun 19 '19 15:06 neolit123

does the kubelet allow self-taints with node roles such as node-role.kubernetes.io/master=:NoSchedule?

Yes, probably we can keep the phase to add the label

Jun 19 '19 16:06 yagonobre

@neolit123 : Can I take a crack at this?

Aug 26 '19 05:08 madhukar32

@madhukar32 hi, this needs discussion before sending the PR. if we remove the taint from markcontrolplane phase it will break existing users.

Aug 26 '19 12:08 neolit123

I was away, but now I'll have enough time to do this. @neolit123 what you think about add the taint to the kubelet config, and keep it on the markcontrolplane phase just for backward compatibility?

Aug 26 '19 20:08 yagonobre

we might have to keep it in both places with a deprecation notice.

Aug 26 '19 21:08 neolit123

We are seeing this when joining control plane nodes to existing clusters (1.15). We use nginx-ingress-controller as a daemonset, and it's on host port 443, same as the apiserver. So the apiserver always ends up in a CrashLoop until I manually delete the pod.

Dec 05 '19 09:12 blurpy

this is a known problem. changes in kubeadm phases are tricky - the existing workaround can be seen above, but we might have to have a period of time where we both taint using the kubelet configuration and the kubeadm mark-control-plane phase, potentially deprecating the tainting in the phase in the future.

Dec 05 '19 13:12 neolit123

Not sure I understand how the workaround works. Isn't InitConfiguration used only during init of the first master? Or can it be updated in the configmap in kube-system and used during join --control-plane?

Dec 05 '19 14:12 blurpy

both init and join configurations have the node registration options: https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2#NodeRegistrationOptions

tainting using the KubeletConfigurtation is not possible.

Dec 05 '19 14:12 neolit123

I've tried to use a JoinConfiguration to setup additional masters, but then I just get this message:

# kubeadm join --control-plane --config /etc/kubernetes/kubeadm-master-config.yaml
     can not mix '--config' with arguments [control-plane]

Related issue I found: #1485

Dec 05 '19 14:12 blurpy

Today I add the taint directly to the kubelet config file. I'll try to work on it soon.

Dec 05 '19 16:12 yagonobre

Yago, i dont see a field for that in the kubeletconfiguration.

Dec 05 '19 16:12 neolit123

I've tried to use a JoinConfiguration to setup additional masters, but then I just get this message:

The join configuration can do that. Some flags and config cannot be mixed.

Dec 05 '19 16:12 neolit123

Yago, i dont see a field for that in the kubeletconfiguration.

Sorry, it's a flag. I use KUBELET_EXTRA_ARGS="--register-with-taints=node-role.kubernetes.io/master=:NoSchedule"

Dec 05 '19 18:12 yagonobre

@yagonobre i just tied passing --register-with-taints=node-role.kubernetes.io/master:NoSchedule to the kubelet instead of using the markcontorlplane phase for the taint and both the CNI (calico) and coredns are stuck in Pending saying that there is no Node to schedule on (even if they tolerate the master taint).

what k8s version have you tried this with? i'm testing with the latest 1.18.

Dec 05 '19 22:12 neolit123

I'm using v1.16.2, but I'll try with the last version.

On Thu, Dec 5, 2019 at 7:59 PM Lubomir I. Ivanov [email protected] wrote:

@yagonobre https://github.com/yagonobre i just tied passing --register-with-taints=node-role.kubernetes.io/maste r:NoSchedule to the kubelet instead of using the markcontorlplane phase for the taint and both the CNI and coredns are stuck in Pending saying that there is no Node to schedule on (even if they tolerate the master taint).

what k8s version have you tried this with? i'm testing with the latest 1.18.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/kubeadm/issues/1621?email_source=notifications&email_token=ACJ5C2YRJKOV45LD5QDMOBTQXGBVDA5CNFSM4HZKKCA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGCOCCQ#issuecomment-562356490, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJ5C24GER64KYQH5XHTMRDQXGBVDANCNFSM4HZKKCAQ .

Dec 05 '19 23:12 yagonobre

@yagonobre never mind. i made the mistake of deploying a kubelet much newer than the kube-apiserver -> 1.18 vs 1.16.3.

will send a POC PR that fixes this in a bit. although it needs discussion.

Dec 05 '19 23:12 neolit123

here is the POC https://github.com/kubernetes/kubernetes/pull/85977

Dec 05 '19 23:12 neolit123

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.