[support request] The Compatibility Problem of Koordinator in KIND Scenario.

[ZiMengSheng]

What is your problem or scenario? The Compatibility Problem of Koordinator in KIND Scenario. Procedure step 1. Install docker, KIND in local MAC machine. step 2. Create a cluster by kind. step 3. Install Koordinator in the cluster created in step2. Problem

1. We can discover that koordlet is always restarting, as shown in figure：

2. check logs of this pod：

3. we can see that koordelet can't detect kubelet cgroup driver and the error log location is koordlet.go line 87 and line 83:

[jasonliu747]

/assign @saintube PTAL

[saintube]

@ZiMengSheng Could you please show us the output when executing the commands below?

$ # execute on a KinD node
$ ls -l /sys/fs/cgroup/cpu
$ ps -ef | grep "kubelet"

[ZiMengSheng]

@ZiMengSheng Could you please show us the output when executing the commands below?

$ # execute on a KinD node
$ ls -l /sys/fs/cgroup/cpu
$ ps -ef | grep "kubelet"

@saintube Okay. The output of command ls -l /sys/fs/cgroup/cpu is as follows: . The output of command ps -ef | grep "kubelet" is as follows: .

[saintube]

@ZiMengSheng It seems the KinD node has an abnormal configuration of Linux CGroups, which koordlet expects to set as /sys/fs/cgroup by default. Please check the real cgroup root of the KinD node and try to modify the volumeMount of koordlet host-cgroup.

# old template for DaemonSet koordlet
     ...
      volumes:
        - hostPath:
            path: /etc/localtime
            type: ""
          name: host-time
        - hostPath:
            path: /sys/fs/cgroup/
            type: ""
          name: host-cgroup-root

# new template for DaemonSet koordlet
     ...
      volumes:
        - hostPath:
            path: /etc/localtime
            type: ""
          name: host-time
        - hostPath:
            path: $real_cgroup_root
            type: ""
          name: host-cgroup-root

[ZiMengSheng]

This is the contents of /sys/fs/cgroup: I don't have another linux machine, could show me normal contents of /sys/fs/cgroup in normal linux machine.

[saintube]

@ZiMengSheng OK... Now I know the real reason.
KinD nodes enables cgroups-v2, while koordlet currently requires cgroups-v1 system. Could you please set cgroups-v1 in KinD nodes so the koordlet would identify the cgroups paths correctly?

[ZiMengSheng]

I have set cgroups-v1 in KinD node. But koordlet doesn't identify cgroups paths. this is the contents of /sys/fs/cgroup/cpu.

[saintube]

@ZiMengSheng It seems KinD also changed the cgroups root configuration of kubelet. Could you please check the CgroupRoot configured in your KinD nodes.
FYI: https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/

e.g.

grep -i "cgroup" /var/lib/kubelet/config.yaml

[ZiMengSheng]

@saintube this is the output of grep -i "cgroup" /var/lib/kubelet/config.yaml

cgroupDriver: systemd
cgroupRoot: /kubelet

but I can't cd into /kubelet

[saintube]

@saintube this is the output of grep -i "cgroup" /var/lib/kubelet/config.yaml

cgroupDriver: systemd
cgroupRoot: /kubelet

but I can't cd into /kubelet

@ZiMengSheng I think cgroupRoot represents a sub-path of the real kubelet cgroup root. So could you please check the sub directories of your /sys/fs/cgroup/kubelet.slice or /sys/fs/cgroup/kubelet. Then you may get a cgroup path namedkubepods.slice, which is the wanted cgroup root of kubelet.

[ZiMengSheng]

yes, you may be right. this is the path similar to your cgrouppath, but named kubelet-kubepods.slice.

[saintube]

yes, you may be right. this is the path similar to your cgrouppath, but named kubelet-kubepods.slice.

@ZiMengSheng So KinD uses a custom CgroupRoot for the Kubelet, which Koordlet does not support. Currently, Koordlet uses the default cgroup path of Kubelet since we often keep this configuration unchanged. This is a compatibility issue and Koordlet needs to support custom cgroup root, WDYT @zwzhang0107 @jasonliu747 @eahydra

[jasonliu747]

I assume kind is for local development environment only, not production use. And yes, we definitely can support it in the future release. But right now, I think the best for us is to add it into known issues list.

[zwzhang0107]

current koordlet only support cgroup v1 cgroup path configuration has been supported, consider adding a feature-gate for KinD to support cgroup path prefix

[jasonliu747]

/close Let's track this in #407

[koordinator-bot[bot]]

@jasonliu747: Closing this issue.

In response to this:

/close Let's track this in #407

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.