proxmox-csi-plugin
proxmox-csi-plugin copied to clipboard
Published
20 hours ago •
sergelogvinov
sergelogvinov
node-role.kubernetes.io/control-plane unset on k3s worker nodes
Bug Report
Description
This is less of a bug report than a question, I suppose.
On my k3s cluster, the node-role.kubernetes.io/control-plane label is not set on the worker nodes. It is set to true on the manager nodes.
Without node-role.kubernetes.io/control-plane: "" set in the worker nodes, the controller pod doesn't come up. If I add this parameter to the worker nodes, the pod gets scheduled, but I'm not clear on what impact this has / does.
My kubernetes knowledge is not deep enough to understand if there's a difference between node-role.kubernetes.io/control-plane: "" and having the label outright unset on the node?
Logs
Controller: [kubectl logs -c proxmox-csi-plugin-controller proxmox-csi-plugin-controller-...]
No logs available; the controller pod is stuck in a pending state:
➜ ~ kubectl -n csi-proxmox describe pod proxmox-csi-plugin-controller-7d79cfbbdd-2bl8b
// snip
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m40s default-scheduler 0/7 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/master: true}, 4 node(s) didn't match Pod's node affinity/selector. preemption: 0/7 nodes are available: 7 Preemption is not helpful for scheduling.
Node: [kubectl logs -c proxmox-csi-plugin-node proxmox-csi-plugin-node-...]
➜ ~ kubectl -n csi-proxmox logs proxmox-csi-plugin-node-f9gwq
Defaulted container "proxmox-csi-plugin-node" out of: proxmox-csi-plugin-node, csi-node-driver-registrar, liveness-probe
I0525 04:28:33.946218 1 main.go:54] Driver version 0.4.0, GitVersion edge, GitCommit 76c899e
I0525 04:28:33.946261 1 main.go:55] Driver CSI Spec version: 1.9.0
I0525 04:28:33.946266 1 main.go:83] Building kube configs for running in cluster...
I0525 04:28:33.947957 1 mount_linux.go:288] Detected umount with safe 'not mounted' behavior
I0525 04:28:33.947995 1 main.go:140] Listening for connection on address: &net.UnixAddr{Name:"/csi/csi.sock", Net:"unix"}
I0525 04:28:34.069053 1 identity.go:38] GetPluginInfo: called
I0525 04:28:34.194704 1 identity.go:38] GetPluginInfo: called
I0525 04:28:34.306598 1 node.go:532] NodeGetInfo: called with args {}
Environment
- Plugin version: Applied from the master branch (don't know this answer)
- Kubernetes version: Server Version: v1.29.2+k3s1
- CSI capasity:
➜ ~ kubectl get csistoragecapacities -ocustom-columns=CLASS:.storageClassName,AVAIL:.capacity,ZONE:.nodeTopology.matchLabels -A
CLASS AVAIL ZONE
proxmox-data-xfs 0 map[topology.kubernetes.io/region:hogwarts topology.kubernetes.io/zone:proxmox2]
proxmox-data-xfs 0 map[topology.kubernetes.io/region:hogwarts topology.kubernetes.io/zone:proxmox4]
proxmox-data-xfs 0 map[topology.kubernetes.io/region:hogwarts topology.kubernetes.io/zone:proxmox3]
proxmox-data-xfs 0 map[topology.kubernetes.io/region:hogwarts topology.kubernetes.io/zone:proxmox1]
proxmox-data 0 map[topology.kubernetes.io/region:hogwarts topology.kubernetes.io/zone:proxmox1]
proxmox-data 0 map[topology.kubernetes.io/region:hogwarts topology.kubernetes.io/zone:proxmox3]
proxmox-data 0 map[topology.kubernetes.io/region:hogwarts topology.kubernetes.io/zone:proxmox2]
proxmox-data 0 map[topology.kubernetes.io/region:hogwarts topology.kubernetes.io/zone:proxmox4]
- CSI resource on the node:
➜ ~ kubectl get CSINode k3s-n1 -oyaml
apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
annotations:
storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/vsphere-volume
creationTimestamp: "2024-05-21T01:34:17Z"
name: k3s-n1
ownerReferences:
- apiVersion: v1
kind: Node
name: k3s-n1
uid: 4f75c5b6-bf4e-41cc-94d6-42aa0079ae4d
resourceVersion: "1872002"
uid: 98389734-7627-45e9-aa01-382e11aaec12
spec:
drivers:
- name: local.csi.openebs.io
nodeID: k3s-n1
topologyKeys:
- kubernetes.io/hostname
- openebs.io/nodename
- name: zfs.csi.openebs.io
nodeID: k3s-n1
topologyKeys:
- beta.kubernetes.io/arch
- beta.kubernetes.io/instance-type
- beta.kubernetes.io/os
- kubernetes.io/arch
- kubernetes.io/hostname
- kubernetes.io/os
- node.kubernetes.io/instance-type
- openebs.io/csi-node
- openebs.io/csi-node.nvme-ana
- openebs.io/engine
- openebs.io/nodeid
- openebs.io/nodename
- topology.kubernetes.io/region
- topology.kubernetes.io/zone
- allocatable:
count: 24
name: csi.proxmox.sinextra.dev
nodeID: k3s-n1
topologyKeys:
- topology.kubernetes.io/region
- topology.kubernetes.io/zone
- Node describe:
➜ ~ kubectl describe node k3s-n1
Name: k3s-n1
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=k3s
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=k3s-n1
kubernetes.io/os=linux
node.kubernetes.io/instance-type=k3s
openebs.io/csi-node=mayastor
openebs.io/csi-node.nvme-ana=true
openebs.io/engine=mayastor
openebs.io/nodeid=k3s-n1
openebs.io/nodename=k3s-n1
topology.kubernetes.io/region=hogwarts
topology.kubernetes.io/zone=proxmox1
Annotations: alpha.kubernetes.io/provided-node-ip: 192.168.102.61
csi.volume.kubernetes.io/nodeid: {"csi.proxmox.sinextra.dev":"k3s-n1","local.csi.openebs.io":"k3s-n1","zfs.csi.openebs.io":"k3s-n1"}
flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"0e:60:36:cc:96:29"}
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 192.168.102.61
k3s.io/hostname: k3s-n1
k3s.io/internal-ip: 192.168.102.61
k3s.io/node-args:
["agent","--server","https://192.168.102.10:6443","--token","********","--flannel-iface","eth0","--node-ip","192.168.102.61"]
k3s.io/node-config-hash: 35C2OIGCH3GBNJCA6UW4ETEEWY77NCN5YSXSA6IVGG4TDSANUM5A====
k3s.io/node-env: {"K3S_DATA_DIR":"/var/lib/rancher/k3s/data/7d0aa19ffc230d4322f04d1ae8783e54ce189dfc4cbfa0a6afcdcabec2346d0c"}
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 20 May 2024 19:34:17 -0600
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: k3s-n1
AcquireTime: <unset>
RenewTime: Sat, 25 May 2024 06:44:56 -0600
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Sat, 25 May 2024 06:42:23 -0600 Fri, 24 May 2024 22:15:30 -0600 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Sat, 25 May 2024 06:42:23 -0600 Fri, 24 May 2024 22:15:30 -0600 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Sat, 25 May 2024 06:42:23 -0600 Fri, 24 May 2024 22:15:30 -0600 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Sat, 25 May 2024 06:42:23 -0600 Fri, 24 May 2024 22:15:30 -0600 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 192.168.102.61
Hostname: k3s-n1
Capacity:
cpu: 4
ephemeral-storage: 20470152Ki
hugepages-2Mi: 4Gi
memory: 16393472Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 19913363850
hugepages-2Mi: 4Gi
memory: 12199168Ki
pods: 110
System Info:
Machine ID: 5e55139dee83424ea106dba23b812faf
System UUID: 5e55139d-ee83-424e-a106-dba23b812faf
Boot ID: 18f171d1-4169-48f1-a0ea-bafb11fa945c
Kernel Version: 6.1.0-21-cloud-amd64
OS Image: Debian GNU/Linux 12 (bookworm)
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.7.11-k3s2
Kubelet Version: v1.29.2+k3s1
Kube-Proxy Version: v1.29.2+k3s1
PodCIDR: 10.42.3.0/24
PodCIDRs: 10.42.3.0/24
ProviderID: k3s://k3s-n1
Non-terminated Pods: (9 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
cert-manager cert-manager-88dd6c6b9-frflh 0 (0%) 0 (0%) 0 (0%) 0 (0%) 4d11h
cert-manager cert-manager-webhook-7ddfd7c4bd-9hsbj 0 (0%) 0 (0%) 0 (0%) 0 (0%) 4d11h
csi-proxmox proxmox-csi-plugin-node-hx5r7 20m (0%) 0 (0%) 32Mi (0%) 0 (0%) 8h
metallb-system speaker-5f6ct 0 (0%) 0 (0%) 0 (0%) 0 (0%) 4d11h
openebs openebs-lvm-localpv-node-mbc86 0 (0%) 0 (0%) 0 (0%) 0 (0%) 12h
openebs openebs-zfs-localpv-controller-6f879b95b-4v67d 0 (0%) 0 (0%) 0 (0%) 0 (0%) 12h
openebs openebs-zfs-localpv-node-6vfkh 0 (0%) 0 (0%) 0 (0%) 0 (0%) 12h
sonarr sonarr-76b9fbc4c8-2lhjk 0 (0%) 0 (0%) 0 (0%) 0 (0%) 4d11h
tt-rss ttrss-dbcluster-3 0 (0%) 0 (0%) 0 (0%) 0 (0%) 12h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 20m (0%) 0 (0%)
memory 32Mi (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events: <none>
- OS version
cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
