NetApp
Trident operator and controller are deployed on a worker node
Describe the bug Operators should be deployed on control-plane nodes, but Trident operator and controller are deployed on a worker node.
$ oc get pod -n trident -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
trident-controller-575f5cb7d4-j8lvh 6/6 Running 0 22d 10.131.1.191 worker-y05by <none> <none>
trident-node-linux-k6rgf 2/2 Running 1 (22d ago) 22d 192.168.30.2 worker-y05by <none> <none>
trident-node-linux-kw6n4 2/2 Running 1493 (20h ago) 21d 192.168.30.3 worker-btov3 <none> <none>
trident-node-linux-qml8t 2/2 Running 1 (22d ago) 22d 192.168.30.7 controller-pagau <none> <none>
trident-node-linux-tq2sf 2/2 Running 1 (22d ago) 22d 192.168.30.5 controller-t2o32 <none> <none>
trident-node-linux-vpfvc 2/2 Running 1 (22d ago) 22d 192.168.30.6 controller-ke1nf <none> <none>
trident-operator-7b46b5b986-lx4jh 1/1 Running 0 21d 10.131.1.192 worker-y05by <none> <none>
Environment
- Trident version: 24.10.0
- Kubernetes orchestrator: OpenShift v4.16
To Reproduce Allways after installation of trident-operator on OpenShift.
I tried to specify controllerPluginNodeSelector in TridentOrchestrator CRD, but it causes the trident-controller pod becomes pendint state.
apiVersion: trident.netapp.io/v1
kind: TridentOrchestrator
metadata:
name: trident
spec:
debug: false
namespace: trident
silenceAutosupport: true
controllerPluginNodeSelector:
nodetype: master
$ oc get pod -n trident
NAME READY STATUS RESTARTS AGE
trident-controller-95bc5645f-ljql7 0/6 Pending 0 108s
...
$ oc describe pod trident-controller-95bc5645f-ljql7 -n trident
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 50s default-scheduler 0/5 nodes are available: 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.
Expected behavior Most of other operators are deployed on control-plane nodes without specifying nodeSelector like:
$ oc get pod -n nvidia-network-operator -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nvidia-network-operator-controller-manager-9fccfb57f-d8jsd 1/1 Running 0 3d22h 10.129.0.43 controller-pagau <none> <none>
Following document site explains that trident-controller can be deplyed on master (i.e., control-plane) nodes. https://docs.netapp.com/us-en/trident/trident-get-started/kubernetes-customize-deploy.html#sample-configurations
apiVersion: trident.netapp.io/v1
kind: TridentOrchestrator
metadata:
name: trident
spec:
debug: true
namespace: trident
controllerPluginNodeSelector:
nodetype: master
nodePluginNodeSelector:
storage: netapp
Hi, For trident-operator deployment, you need to set this two values (based on your error message, maybe it needs an etcd toleration too):
nodeSelector:
nodetype: master
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
Same for trident-controller deployment with tridentControllerPluginNodeSelector and tridentControllerPluginTolerations
Hi enneitex, Thanks for your comment.
I tried to change TridentOrchestrator CRD to:
apiVersion: trident.netapp.io/v1
kind: TridentOrchestrator
metadata:
name: trident
spec:
debug: false
namespace: trident
silenceAutosupport: true
controllerPluginNodeSelector:
nodetype: master
controllerPluginTolerations:
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
ref: Customize Trident operator installation
As the result, the controller pod is still pending state.
$ oc describe pod trident-controller-56c4b48cc8-wmjhr -n trident
Name: trident-controller-56c4b48cc8-wmjhr
Namespace: trident
Priority: 0
Service Account: trident-controller
Node: <none>
Labels: app=controller.csi.trident.netapp.io
pod-template-hash=56c4b48cc8
Annotations: openshift.io/required-scc: trident-controller
openshift.io/scc: trident-controller
Status: Pending
...
Node-Selectors: <none>
Tolerations: node-role.kubernetes.io/master:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 34s default-scheduler 0/5 nodes are available: 5 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.
Control-plane nodes have a taint of node-role.kubernetes.io/master:NoSchedule only.
$ oc describe node controller-ke1nf
Name: controller-ke1nf
Roles: control-plane,master
...
Taints: node-role.kubernetes.io/master:NoSchedule
...
On the other hand, Trident Operator is automatically deployed by OLM on OpenShift with Subscription resource.
For example of nvidia-network-operator listed above, there are some specifications for affinity in ClusterServiceVersion (related to #980).
Trident operator doesn't have any specifications for affinity in its CSV.
$ oc describe csv nvidia-network-operator -n nvidia-network-operator
Name: nvidia-network-operator.v24.10.1
...
Deployments:
Label:
Control - Plane: nvidia-network-operator-controller
Name: nvidia-network-operator-controller-manager
Spec:
Replicas: 1
Selector:
Match Labels:
Control - Plane: nvidia-network-operator-controller
Strategy:
Template:
Metadata:
Annotations:
kubectl.kubernetes.io/default-container: manager
Creation Timestamp: <nil>
Labels:
Control - Plane: nvidia-network-operator-controller
nvidia.com/ofed-driver-upgrade-drain.skip: true
Spec:
Affinity:
Node Affinity:
Preferred During Scheduling Ignored During Execution:
Preference:
Match Expressions:
Key: node-role.kubernetes.io/master
Operator: In
Values:
Weight: 1
Preference:
Match Expressions:
Key: node-role.kubernetes.io/control-plane
Operator: In
Values:
Weight: 1
...
Are you sure the label nodetype: master exists on your control-planes ? If not change controllerPluginNodeSelector and nodeSelector to match an existing one
There is no nodetype: master label. I didn't check it. (I just follow the sample)
Even if controllerPluginNodeSelector is omitted, the results are same.
I deleted the previous TridentOrchestrator and re-applyed following YAML, but the result was same.
apiVersion: trident.netapp.io/v1
kind: TridentOrchestrator
metadata:
name: trident
spec:
debug: false
namespace: trident
silenceAutosupport: true
controllerPluginTolerations:
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
$ oc describe pod trident-controller-56c4b48cc8-4khff -n trident
...
Node-Selectors: <none>
Tolerations: node-role.kubernetes.io/master:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m35s default-scheduler 0/5 nodes are available: 5 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.
$ oc describe node controller-ke1nf
Name: controller-ke1nf
Roles: control-plane,master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=controller-ke1nf
kubernetes.io/os=linux
node-role.kubernetes.io/control-plane=
node-role.kubernetes.io/master=
node.openshift.io/os_id=rhcos
...
Taints: node-role.kubernetes.io/master:NoSchedule
@enneitex , just to clarify for everyone: is putting the operator to the control plane nodes the recommended setup? If yes is this documented somewhere?
I don't know what's the recommended setup.
@kimuraos sorry I suggested nodetype: master because it was written in your first message.
Based on the describe output, you should set this:
apiVersion: trident.netapp.io/v1
kind: TridentOrchestrator
metadata:
name: trident
spec:
debug: false
namespace: trident
silenceAutosupport: true
controllerPluginNodeSelector:
node-role.kubernetes.io/control-plane: ""
controllerPluginTolerations:
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
and
nodeSelector:
node-role.kubernetes.io/control-plane: ""
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
You should end up with trident-controller and trident-operator both on control-planes. I'm not familiar with openshift and ClusterServiceVersion sorry.
Unfortunately, the cluster was torn down for another test. I can't re-test with the cluster.
I'm not sure about recommendation of operator placement, but most of operators are placed in one of control-plane nodes on OpenShift. As I described above, trident-operator is deployed by OLM (Operator Lifecycle Manager) on OpenShift. There is no way to specify nodeSelector and/or tolerations at initial deployment, from my understanding. Patch may be able to apply to its deployment, but I think Trident's ClusterServiceVersion should be ammended.
On the other hand, trident-controller is original component of Trident. There is no generic recommendation of placement, but I think it should be placed in one of control-plane nodes as its function (similar to operator). I just tried to follow sample. I expect the web site to replace to more suitable example.
@kimuraos Has your issue since been resolved? Typically it is fine to run the Trident operator and controller on a worker node. Trident doesn't need to be running on the control-plane.
As I stated, the cluster was torn down. No further tests were re-taken. I know there is no problem to run the Trident operator and controller on a worker node, but it is a fact that many other operators are scheduled on a control-plane node as default under OpenShift environment.