Allow overwrite KubeletRunDirectory when init/join

[pytimer]

Is this a BUG REPORT or FEATURE REQUEST?

FEATURE REQUEST

Versions

kubeadm version (use kubeadm version): v1.17.2

Environment:

• Kubernetes version (use kubectl version): v1.17.2
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

What happened?

I want to change the kubelet run directory when init or join nodes, setting root-dir in kubeadm kubeletExtraArgs field of nodeRegistration object. But kubeadm also write the kubelet config.yaml to /var/lib/kubelet and this behavior make init fail.

What you expected to happen?

I hope kubeadm can setting KubeletRunDirectory. I see this variable is constant now.

The code: https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/constants/constants.go#L244

I want to know my idea wether correct, i see a relate issue: https://github.com/kubernetes/kubeadm/issues/1478 . If needs, i will try to do it.

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

[neolit123]

see here: https://github.com/kubernetes/kubeadm/issues/1478#issuecomment-484170978

you can solve this with a symlink. last time we spoke about this we didn't want to enable the override on the side of kubeadm.

/priority awaiting-more-evidence /kind feature

[pytimer]

@neolit123 Thanks for reply.

Use symlink can solve my problem. But in my case, i use ansible to deploy some clusters, some of them use /var/lib/kubelet, but another use other directory, if use symlink, i should compare with custom kubelet run directory and /var/lib/kubelet, and decide wether use symlink. This adds a little complexity, so if kubeadm support, it make me easy to do it.

[neolit123]

potentially, instead of writing the config always to the default kubelet directory, we can check if the user has passed a KubeletConfiguration and use the value from there. unfortunately this means on joining nodes we have to use flags only as kubeadm join currently does not support KubeletConfiguration. so that's a much larger complexity on the kubeadm side.

maybe the kubeadm API should support the field (e.g. in NodeRegistrationOptions), instead of making assumption based on KubeletConfiguration.

[neolit123]

/kind design

[fabriziopandini]

I'm +1 to defer this to when the design for node specific settings in component config is complete. We are trying to avoid to create/maintain knobs for specific flags

[pytimer]

If the issue https://github.com/kubernetes/kubeadm/issues/1682 solved, i think this issue also solved.

[neolit123]

related KEP https://github.com/kubernetes/enhancements/pull/1439

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[neolit123]

/remove-lifecycle stale let's re-evaluate again after https://github.com/kubernetes/enhancements/pull/1439

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[fabriziopandini]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[neolit123]

/remove-lifecycle stale

[pacoxu]

• P3: customize /var/lib/kubelet

I think we can add KubeletRootDir in v1beta3 like CertificatesDir. I would work on it if there is no objection. /assign

[neolit123]

Is kubeadm writing anything to /var/run/kubelet? If not I do not think we should have it in the kubeadm api.

We do write to /var/lib/kubelet. The tricky part would be to sync this dir between kubeadm config and kubelet config.

[pacoxu]

  --root-dir string  Directory path for managing kubelet files (volume mounts,etc). (default "/var/lib/kubelet")

I mean /var/lib/kubelet.

[neolit123]

@pacoxu @pytimer

we had a discussion with @fabriziopandini about configurable /var/lib/kubelet. the main problem here is that if we allow this to be configurable it must be in ClusterConfiguration, because Init and JoinConfiguration are not used during upgrade. /var/lib/kubelet is used during upgrade to write the kubelet.conf file on disk that contains the KubeletConfiguration.

at the same time we don't want to add this field to ClusterConfiguration, because in the future we want to allow the users to have a fully configurable node specific configuration.

because of this complexity we don't want to add this change in v1beta3 / 1.22 and i'm going to have to move the milestone to Next. in the future we hope there will be a way to persist some of the user settings for a node either on disk or in the cluster.

[neolit123]

@pacoxu if you want to help with v1beta3 you can take: https://github.com/kubernetes/kubeadm/issues/524

[pytimer]

the main problem here is that if we allow this to be configurable it must be in ClusterConfiguration, because Init and JoinConfiguration are not used during upgrade. /var/lib/kubelet is used during upgrade to write the kubelet.conf file on disk that contains the KubeletConfiguration.

Maybe the every node kubelet configuration not the same in the cluster, this parameter is not suitable in ClusterConfiguration.

Now i use configSource to setting every node kubelet configuration. It like https://github.com/kubernetes/enhancements/pull/1439.

spec:
  configSource:
    configMap:
      kubeletConfigKey: kubelet
      name: kubelet-configmap-master-101
      namespace: kube-system

Can use the kubelet configSource to do it?

[neolit123]

We have not yet decided how to handle instance specific confinguration in kubeadm. It is a complicated topic.

[pacoxu]

/unassign since not yet decided.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[neolit123]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[chendave]

Early investigation for this issue, might not accurate enough, correct me pls.

• the original requirement for this issue is the support of the overwrite the kubelet run directory, and this is still hard-coded today,

	// KubeletRunDirectory specifies the directory where the kubelet runtime information is stored.
	KubeletRunDirectory = "/var/lib/kubelet"

• This issue is pending because there was node specific config feature: https://github.com/kubernetes/enhancements/pull/1439, but it is too complicated, there is no plan to revive that feature.

• the feature of patch that was added https://github.com/kubernetes/kubernetes/pull/110405 cannot address this issue, since it's just patch the existing kubelet config, instead of make the kubelet run directory configurable.

So, this issue still stands, and one tradeoff is making KubeletRunDirectory a config item in the clusterConfiguration, I know this was pushed back because there was node-specific config feature considered at that time.

@neolit123 @pacoxu @SataQiu Can we include this in v1beta4? one day the node-specific config is implemented, we can pull this out from clusterConfiguration then, WDYT?

[neolit123]

+1 to remove from cluster scope. it can be a feature in noderegistrationoptions, but i think the demand is relatively low. users can use symlinks too, i think.

[pacoxu]

We need to be careful with it.

As https://github.com/kubernetes/kubeadm/issues/2104#issuecomment-844170913 pointed out, kubelet.conf file will place based on the kubelet root-dir. Both config.yaml and kubeadm-flags.env. This should be included in the docs.

BTW, kubelet cert-dir is not using the root-dir, the user should specify it as well to change the cert-dir when needed.

--cert-dir string     Default: /var/lib/kubelet/pki

+1 to remove from cluster scope. it can be a feature in noderegistrationoptions, but I think the demand is relatively low. users can use symlinks too, i think.

https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta3/#kubeadm-k8s-io-v1beta3-NodeRegistrationOptions

A question when I implement https://github.com/kubernetes/enhancements/pull/3930: use kubeletExtraArgs will be combined with the cirSocket to save into the kubelet env file.

https://github.com/kubernetes/kubernetes/blob/01b6bb41b0b912535025e82028d0f180a0416e02/cmd/kubeadm/app/phases/kubelet/flags.go#L60-L73

~~If we add root-dir support, it would be similar to the criSocket. (not best practice IMO, though this may have a historical reason that we want to be consistent to the --cri-socket flag.)~~ EDITED

[neolit123]

We need to be careful with it.

agreed, i think we should just drop it from the cluster configuration as a start. implementation / piping the option in kubeadm feels google doc proposal worthy.

[neolit123]

A question when I implement kubernetes/enhancements#3930: use kubeletExtraArgs will be combined with the cirSocket to save into the kubelet env file.

the kep in question should be using the kubeletconfiguration to set a local criSocket, maybe by storing a patch file in the kubelet dir, no? (i think i commented about this idea already there)

i don't think we should use kubelet flags for new features.