Unable to deploy v0.72.0 in OKD 4.10.0-0.okd-2022-07-09-073606

[JoseManuelPS]

What happened: I have tried to deploy the new version v0.72.0 in an OKD cluster, the operator is deployed correctly but the pod never gets generated. This is probably due to a bug in the security context configuration.

What you expected to happen: The replicaset manages to deploy the desired pod.

How to reproduce it (as minimally and precisely as possible): I have started with a completely clean okd cluster and deployed the operator:

kubectl apply -f https://github.com/nmstate/kubernetes-nmstate/releases/download/v0.72.0/nmstate.io_nmstates.yaml
kubectl apply -f https://github.com/nmstate/kubernetes-nmstate/releases/download/v0.72.0/namespace.yaml
kubectl apply -f https://github.com/nmstate/kubernetes-nmstate/releases/download/v0.72.0/service_account.yaml
kubectl apply -f https://github.com/nmstate/kubernetes-nmstate/releases/download/v0.72.0/role.yaml
kubectl apply -f https://github.com/nmstate/kubernetes-nmstate/releases/download/v0.72.0/role_binding.yaml
kubectl apply -f https://github.com/nmstate/kubernetes-nmstate/releases/download/v0.72.0/operator.yaml
kubectl apply -f https://github.com/nmstate/kubernetes-nmstate/releases/download/v0.72.0/scc.yaml

Anything else we need to know?: I also tried to deploy the operator from operatorhub, but that didn't work either. With this method the pod manages to deploy, but it constantly reproduces the following failure after create a nmstate instance.

E0715 15:37:53.481574 1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:openshift-nmstate:nmstate-operator" cannot list resource "nodes" in API group "" at the cluster sco

Environment:

• kubernetes-nmstate image (use kubectl get pods --all-namespaces -l app=kubernetes-nmstate -o jsonpath='{.items[0].spec.containers[0].image}'):

$ kubectl get deployment nmstate-operator -o=jsonpath='{.spec.template.spec.containers[0].image}'
quay.io/nmstate/kubernetes-nmstate-operator:v0.72.0

• Kubernetes version (use kubectl version): v1.23.5+3afdacb - OKD 4.10.0-0.okd-2022-07-09-073606
• Others:
• Logs from replicaset:

: Forbidden: not usable by user or serviceaccount provider "hostaccess": Forbidden: not usable by user or serviceaccount provider "nmstate": Forbidden: not usable by user or serviceaccount provider "node-exporter": Forbidden: not usable by user or serviceaccount provider "privileged": Forbidden: not usable by user or serviceaccount]
  Warning  FailedCreate  36m                   replicaset-controller  Error creating: pods "nmstate-operator-7858d5899-r8dpt" is forbidden: unable to validate against any security context constraint: [pod.metadata.annotations.seccomp.security.alpha.kubernetes.io/pod: Forbidden: seccomp may not be set pod.metadata.annotations.container.seccomp.security.alpha.kubernetes.io/nmstate-operator: Forbidden: seccomp may not be set provider "anyuid": Forbidden: not usable by user or serviceaccount provider "nonroot": Forbidden: not usable by user or serviceaccount provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount provider "hostnetwork": Forbidden: not usable by user or serviceaccount provider "hostaccess": Forbidden: not usable by user or serviceaccount provider "nmstate": Forbidden: not usable by user or serviceaccount provider "node-exporter": Forbidden: not usable by user or serviceaccount provider "privileged": Forbidden: not usable by user or serviceaccount]
  Warning  FailedCreate  36m                   replicaset-controller  Error creating: pods "nmstate-operator-7858d5899-9b9ft" is forbidden: unable to validate against any security context constraint: [pod.metadata.annotations.seccomp.security.alpha.kubernetes.io/pod: Forbidden: seccomp may not be set pod.metadata.annotations.container.seccomp.security.alpha.kubernetes.io/nmstate-operator: Forbidden: seccomp may not be set provider "anyuid": Forbidden: not usable by user or serviceaccount provider "nonroot": Forbidden: not usable by user or serviceaccount provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount provider "hostnetwork": Forbidden: not usable by user or serviceaccount provider "hostaccess": Forbidden: not usable by user or serviceaccount provider "nmstate": Forbidden: not usable by user or serviceaccount provider "node-exporter": Forbidden: not usable by user or serviceaccount provider "privileged": Forbidden: not usable by user or serviceaccount]
  Warning  FailedCreate  2m39s (x11 over 36m)  replicaset-controller  (combined from similar events): Error creating: pods "nmstate-operator-7858d5899-jxdrt" is forbidden: unable to validate against any security context constraint: [pod.metadata.annotations.seccomp.security.alpha.kubernetes.io/pod: Forbidden: seccomp may not be set pod.metadata.annotations.container.seccomp.security.alpha.kubernetes.io/nmstate-operator: Forbidden: seccomp may not be set provider "anyuid": Forbidden: not usable by user or serviceaccount provider "nonroot": Forbidden: not usable by user or serviceaccount provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount provider "hostnetwork": Forbidden: not usable by user or serviceaccount provider "hostaccess": Forbidden: not usable by user or serviceaccount provider "nmstate": Forbidden: not usable by user or serviceaccount provider "node-exporter": Forbidden: not usable by user or serviceaccount provider "privileged": Forbidden: not usable by user or serviceaccount]

[phoracek]

Thanks for reporting it.

@qinqon I see that the openshift operator had a similar issue recently, fixed via https://github.com/openshift/kubernetes-nmstate/pull/287. Should we have a similar PR for upstream?

[qinqon]

We have that already https://github.com/nmstate/kubernetes-nmstate/blob/main/deploy/operator/role.yaml#L28-L34

[qinqon]

@creydr could this be related to something missing at the openshift fork ?

[creydr]

Should not be related to the openshift fork, as the manifests from upstreams are used

[qinqon]

Should not be related to the openshift fork, as the manifests from upstreams are used

Maybe the SCC upstream is missing something ?

[creydr]

I could not reproduce the exact same behavior on 4.11.0-0.okd-2022-07-27-013906. When using the 0.72 manifests the operator pod comes up, but the handler pods don't. The issue with the handler pods is addressed in #1113.