Allow must-gather pod to run on master nodes
After running CO's must-gather image, there were no raw-results collected.
[must-gather-9qghh] POD 2024-12-05T13:16:30.192357799Z + oc create -n openshift-compliance -f /must-gather/openshift-compliance/raw-results//extract-pods//must-gather-raw-results-ocp4-cis-node-worker-pod.yaml
[must-gather-9qghh] POD 2024-12-05T13:16:30.201245730Z + CLAIMNAME=ocp4-cis
[must-gather-9qghh] POD 2024-12-05T13:16:30.201289161Z + EXTRACT_POD_NAME=must-gather-raw-results-ocp4-cis
[must-gather-9qghh] POD 2024-12-05T13:16:30.201289161Z + sed s/%CLAIMNAME%/ocp4-cis/g /usr/share/fetch-raw-results-pod-template.yaml
[must-gather-9qghh] POD 2024-12-05T13:16:30.204334081Z + oc create -n openshift-compliance -f /must-gather/openshift-compliance/raw-results//extract-pods//must-gather-raw-results-ocp4-cis-pod.yaml
[must-gather-9qghh] POD 2024-12-05T13:16:30.210048980Z + CLAIMNAME=ocp4-cis-node-master
[must-gather-9qghh] POD 2024-12-05T13:16:30.210728787Z + EXTRACT_POD_NAME=must-gather-raw-results-ocp4-cis-node-master
[must-gather-9qghh] POD 2024-12-05T13:16:30.210728787Z + sed s/%CLAIMNAME%/ocp4-cis-node-master/g /usr/share/fetch-raw-results-pod-template.yaml
[must-gather-9qghh] POD 2024-12-05T13:16:30.213622313Z + oc create -n openshift-compliance -f /must-gather/openshift-compliance/raw-results//extract-pods//must-gather-raw-results-ocp4-cis-node-master-pod.yaml
[must-gather-9qghh] POD 2024-12-05T13:16:30.298190265Z pod/must-gather-raw-results-ocp4-cis-node-worker created
[must-gather-9qghh] POD 2024-12-05T13:16:30.303047951Z + oc wait -n openshift-compliance --for=condition=Ready pod/must-gather-raw-results-ocp4-cis-node-worker
[must-gather-9qghh] POD 2024-12-05T13:16:30.316625885Z pod/must-gather-raw-results-ocp4-cis created
[must-gather-9qghh] POD 2024-12-05T13:16:30.320706001Z + oc wait -n openshift-compliance --for=condition=Ready pod/must-gather-raw-results-ocp4-cis
[must-gather-9qghh] POD 2024-12-05T13:16:30.327363294Z pod/must-gather-raw-results-ocp4-cis-node-master created
[must-gather-9qghh] POD 2024-12-05T13:16:30.330668210Z + oc wait -n openshift-compliance --for=condition=Ready pod/must-gather-raw-results-ocp4-cis-node-master
[must-gather-9qghh] POD 2024-12-05T13:17:00.401868909Z error: timed out waiting for the condition on pods/must-gather-raw-results-ocp4-cis-node-worker
[must-gather-9qghh] POD 2024-12-05T13:17:00.404516399Z + oc cp -n openshift-compliance must-gather-raw-results-ocp4-cis-node-worker:/scan-results /must-gather/openshift-compliance/raw-results//ocp4-cis-node-worker
[must-gather-9qghh] POD 2024-12-05T13:17:00.419921158Z error: timed out waiting for the condition on pods/must-gather-raw-results-ocp4-cis
[must-gather-9qghh] POD 2024-12-05T13:17:00.423023081Z + oc cp -n openshift-compliance must-gather-raw-results-ocp4-cis:/scan-results /must-gather/openshift-compliance/raw-results//ocp4-cis
[must-gather-9qghh] POD 2024-12-05T13:17:00.423783701Z error: timed out waiting for the condition on pods/must-gather-raw-results-ocp4-cis-node-master
[must-gather-9qghh] POD 2024-12-05T13:17:00.426640166Z + oc cp -n openshift-compliance must-gather-raw-results-ocp4-cis-node-master:/scan-results /must-gather/openshift-compliance/raw-results//ocp4-cis-node-master
[must-gather-9qghh] POD 2024-12-05T13:17:00.488990571Z Error from server (BadRequest): pod must-gather-raw-results-ocp4-cis-node-worker does not have a host assigned
[must-gather-9qghh] POD 2024-12-05T13:17:00.492517734Z + oc delete pod -n openshift-compliance must-gather-raw-results-ocp4-cis-node-worker
[must-gather-9qghh] POD 2024-12-05T13:17:00.512846314Z Error from server (BadRequest): pod must-gather-raw-results-ocp4-cis does not have a host assigned
[must-gather-9qghh] POD 2024-12-05T13:17:00.514363714Z Error from server (BadRequest): pod must-gather-raw-results-ocp4-cis-node-master does not have a host assigned
[must-gather-9qghh] POD 2024-12-05T13:17:00.516027558Z + oc delete pod -n openshift-compliance must-gather-raw-results-ocp4-cis
[must-gather-9qghh] POD 2024-12-05T13:17:00.518140814Z + oc delete pod -n openshift-compliance must-gather-raw-results-ocp4-cis-node-master
[must-gather-9qghh] POD 2024-12-05T13:17:00.601800074Z pod "must-gather-raw-results-ocp4-cis-node-worker" deleted
[must-gather-9qghh] POD 2024-12-05T13:17:00.628436251Z pod "must-gather-raw-results-ocp4-cis" deleted
[must-gather-9qghh] POD 2024-12-05T13:17:00.637039309Z pod "must-gather-raw-results-ocp4-cis-node-master" deleted
The pod status shows the following:
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2024-12-05T13:20:21Z"
message: '0/6 nodes are available: 1 node(s) had untolerated taint {node.cloudprovider.kubernetes.io/uninitialized:
true}, 2 node(s) had volume node affinity conflict, 3 node(s) had untolerated
taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available:
6 Preemption is not helpful for scheduling.'
reason: Unschedulable
status: "False"
type: PodScheduled
phase: Pending
qosClass: BestEffort
This aligns with oc-compliance fetch raw-results pods:
https://github.com/openshift/oc-compliance/blob/c46c6947ec8c02c16753746539cb2fa404af189d/internal/fetchraw/compliancescans.go#L335
:robot: To deploy this PR, run the following command:
make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:633-3d4344ceb6e0cb7f78709f8ae86c9390ee553a0a
[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Vincent056, yuumasato
The full list of commands accepted by this bot can be found here.
The pull request process is described here
- ~~OWNERS~~ [Vincent056,yuumasato]
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
@yuumasato I am curious under which condition this issue reproduced. I tried on the last released version compliance-operator.v1.6.1, but I didn't reproduce this issue. Did I miss something? Thanks.
% cd must-gather.local.7183294266625486116
% find ./ -name "*.bzip2"
.//registry-redhat-io-compliance-openshift-compliance-must-gather-rhel8-sha256-81cfb7ecb96e48c08d8cfec0e519df3c8502be11490043a9b0a1aae2e8f8e174/openshift-compliance/raw-results/ocp4-cis/0/ocp4-cis-api-checks-pod.xml.bzip2
.//registry-redhat-io-compliance-openshift-compliance-must-gather-rhel8-sha256-81cfb7ecb96e48c08d8cfec0e519df3c8502be11490043a9b0a1aae2e8f8e174/openshift-compliance/raw-results/ocp4-cis-node-master/0/openscap-pod-595d518f5838bd558782948d451c0aa87ddddfc8.xml.bzip2
.//registry-redhat-io-compliance-openshift-compliance-must-gather-rhel8-sha256-81cfb7ecb96e48c08d8cfec0e519df3c8502be11490043a9b0a1aae2e8f8e174/openshift-compliance/raw-results/ocp4-cis-node-master/0/openscap-pod-e050680f743e3bd99898657a87809ddb93854520.xml.bzip2
.//registry-redhat-io-compliance-openshift-compliance-must-gather-rhel8-sha256-81cfb7ecb96e48c08d8cfec0e519df3c8502be11490043a9b0a1aae2e8f8e174/openshift-compliance/raw-results/ocp4-cis-node-master/0/openscap-pod-d437c405f302eda9e52e7c3a2ebeaee4f68d81d1.xml.bzip2
.//registry-redhat-io-compliance-openshift-compliance-must-gather-rhel8-sha256-81cfb7ecb96e48c08d8cfec0e519df3c8502be11490043a9b0a1aae2e8f8e174/openshift-compliance/raw-results/ocp4-cis-node-worker/0/openscap-pod-7e5b6cad009ef7928265bdd9fa438ffc41f7f785.xml.bzip2
.//registry-redhat-io-compliance-openshift-compliance-must-gather-rhel8-sha256-81cfb7ecb96e48c08d8cfec0e519df3c8502be11490043a9b0a1aae2e8f8e174/openshift-compliance/raw-results/ocp4-cis-node-worker/0/openscap-pod-7e4a20683e13b08d04bce3fedae50df5f53d5160.xml.bzip2
% bzip2 -d .//registry-redhat-io-compliance-openshift-compliance-must-gather-rhel8-sha256-81cfb7ecb96e48c08d8cfec0e519df3c8502be11490043a9b0a1aae2e8f8e174/openshift-compliance/raw-results/ocp4-cis-node-master/0/openscap-pod-595d518f5838bd558782948d451c0aa87ddddfc8.xml.bzip2
bzip2: Can't guess original name for .//registry-redhat-io-compliance-openshift-compliance-must-gather-rhel8-sha256-81cfb7ecb96e48c08d8cfec0e519df3c8502be11490043a9b0a1aae2e8f8e174/openshift-compliance/raw-results/ocp4-cis-node-master/0/openscap-pod-595d518f5838bd558782948d451c0aa87ddddfc8.xml.bzip2 -- using .//registry-redhat-io-compliance-openshift-compliance-must-gather-rhel8-sha256-81cfb7ecb96e48c08d8cfec0e519df3c8502be11490043a9b0a1aae2e8f8e174/openshift-compliance/raw-results/ocp4-cis-node-master/0/openscap-pod-595d518f5838bd558782948d451c0aa87ddddfc8.xml.bzip2.out
% cat .//registry-redhat-io-compliance-openshift-compliance-must-gather-rhel8-sha256-81cfb7ecb96e48c08d8cfec0e519df3c8502be11490043a9b0a1aae2e8f8e174/openshift-compliance/raw-results/ocp4-cis-node-master/0/openscap-pod-595d518f5838bd558782948d451c0aa87ddddfc8.xml.bzip2.out | head
<?xml version="1.0" encoding="UTF-8"?>
<arf:asset-report-collection xmlns:arf="http://scap.nist.gov/schema/asset-reporting-format/1.1" xmlns:core="http://scap.nist.gov/schema/reporting-core/1.1" xmlns:ai="http://scap.nist.gov/schema/asset-identification/1.1">
<core:relationships xmlns:arfvocab="http://scap.nist.gov/specifications/arf/vocabulary/relationships/1.0#">
<core:relationship type="arfvocab:createdFor" subject="xccdf1">
<core:ref>collection1</core:ref>
</core:relationship>
<core:relationship type="arfvocab:isAbout" subject="xccdf1">
<core:ref>asset0</core:ref>
</core:relationship>
</core:relationships>
@yuumasato: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:
| Test name | Commit | Details | Required | Rerun command |
|---|---|---|---|---|
| ci/prow/e2e-rosa | 3d4344ceb6e0cb7f78709f8ae86c9390ee553a0a | link | true | /test e2e-rosa |
| ci/prow/e2e-aws-parallel | 3d4344ceb6e0cb7f78709f8ae86c9390ee553a0a | link | true | /test e2e-aws-parallel |
| ci/prow/e2e-aws-parallel-arm | 3d4344ceb6e0cb7f78709f8ae86c9390ee553a0a | link | true | /test e2e-aws-parallel-arm |
Full PR test history. Your PR dashboard.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.
verication failed. Detailed steps seen from below. @yuumasato Could you please help to take a look? Thanks.
- Deploy Compliance Operator and create a ssb
- Add taint for all nodes
% for node in `oc get node --no-headers | awk '{print $1}'`; do kubectl taint nodes $node node-role.kubernetes.io/master=:NoSchedule --overwrite=true; done
node/ip-10-0-20-178.us-east-2.compute.internal modified
node/ip-10-0-24-124.us-east-2.compute.internal modified
node/ip-10-0-42-65.us-east-2.compute.internal modified
node/ip-10-0-53-194.us-east-2.compute.internal modified
node/ip-10-0-90-177.us-east-2.compute.internal modified
node/ip-10-0-91-136.us-east-2.compute.internal modified
- Check whether the must-gather command works or not:
% oc get csv compliance-operator.v1.6.0 -o yaml | grep -i must-gather
must-gather-image: ghcr.io/complianceascode/must-gather-ocp:latest
% oc adm must-gather --image=ghcr.io/complianceascode/must-gather-ocp:latest
[must-gather ] OUT Using must-gather plug-in image: ghcr.io/complianceascode/must-gather-ocp:latest
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information:
ClusterID: 3ffa5b71-c3da-4045-a897-e744ca12531b
ClientVersion: 4.15.0
ClusterVersion: Stable at "4.18.0-0.nightly-2025-03-01-063647"
ClusterOperators:
All healthy and stable
[must-gather ] OUT namespace/openshift-must-gather-c5jrn created
[must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-45h6g created
...
[must-gather-h7lf5] POD 2025-03-03T09:19:12.502658265Z error: timed out waiting for the condition on pods/must-gather-raw-results-ocp4-stig
[must-gather-h7lf5] POD 2025-03-03T09:19:12.504690794Z + oc cp -n openshift-compliance must-gather-raw-results-ocp4-stig:/scan-results /must-gather/openshift-compliance/raw-results//ocp4-stig
[must-gather-h7lf5] POD 2025-03-03T09:19:12.532576182Z error: timed out waiting for the condition on pods/must-gather-raw-results-rhcos4-stig-master
[must-gather-h7lf5] POD 2025-03-03T09:19:12.534228466Z + oc cp -n openshift-compliance must-gather-raw-results-rhcos4-stig-master:/scan-results /must-gather/openshift-compliance/raw-results//rhcos4-stig-master
[must-gather-h7lf5] POD 2025-03-03T09:19:12.562585272Z error: timed out waiting for the condition on pods/must-gather-raw-results-ocp4-stig-node-worker
[must-gather-h7lf5] POD 2025-03-03T09:19:12.565381757Z + oc cp -n openshift-compliance must-gather-raw-results-ocp4-stig-node-worker:/scan-results /must-gather/openshift-compliance/raw-results//ocp4-stig-node-worker
[must-gather-h7lf5] POD 2025-03-03T09:19:12.585181085Z error: timed out waiting for the condition on pods/must-gather-raw-results-rhcos4-stig-worker
[must-gather-h7lf5] POD 2025-03-03T09:19:12.587068851Z + oc cp -n openshift-compliance must-gather-raw-results-rhcos4-stig-worker:/scan-results /must-gather/openshift-compliance/raw-results//rhcos4-stig-worker
[must-gather-h7lf5] POD 2025-03-03T09:19:12.609590944Z error: timed out waiting for the condition on pods/must-gather-raw-results-ocp4-stig-node-master
[must-gather-h7lf5] POD 2025-03-03T09:19:12.612308132Z + oc cp -n openshift-compliance must-gather-raw-results-ocp4-stig-node-master:/scan-results /must-gather/openshift-compliance/raw-results//ocp4-stig-node-master
[must-gather-h7lf5] POD 2025-03-03T09:19:12.617778761Z Error from server (BadRequest): pod must-gather-raw-results-ocp4-stig does not have a host assigned
[must-gather-h7lf5] POD 2025-03-03T09:19:12.620669284Z + oc delete pod -n openshift-compliance must-gather-raw-results-ocp4-stig
[must-gather-h7lf5] POD 2025-03-03T09:19:12.733448318Z Error from server (BadRequest): pod must-gather-raw-results-rhcos4-stig-master does not have a host assigned
[must-gather-h7lf5] POD 2025-03-03T09:19:12.738007445Z + oc delete pod -n openshift-compliance must-gather-raw-results-rhcos4-stig-master
[must-gather-h7lf5] POD 2025-03-03T09:19:12.763220380Z Error from server (BadRequest): pod must-gather-raw-results-ocp4-stig-node-worker does not have a host assigned
[must-gather-h7lf5] POD 2025-03-03T09:19:12.767239550Z + oc delete pod -n openshift-compliance must-gather-raw-results-ocp4-stig-node-worker
[must-gather-h7lf5] POD 2025-03-03T09:19:12.865857491Z Error from server (BadRequest): pod must-gather-raw-results-ocp4-stig-node-master does not have a host assigned
[must-gather-h7lf5] POD 2025-03-03T09:19:12.869188179Z + oc delete pod -n openshift-compliance must-gather-raw-results-ocp4-stig-node-master
[must-gather-h7lf5] POD 2025-03-03T09:19:12.891004612Z Error from server (BadRequest): pod must-gather-raw-results-rhcos4-stig-worker does not have a host assigned
% oc project openshift-must-gather-c5jrn
Now using project "openshift-must-gather-c5jrn" on server "https://api.xiyuan-18b.qe.devcluster.openshift.com:6443".
% oc get pod must-gather-h7lf5 -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.130.0.198/23"],"mac_address":"0a:58:0a:82:00:c6","gateway_ips":["10.130.0.1"],"routes":[{"dest":"10.128.0.0/14","nextHop":"10.130.0.1"},{"dest":"172.30.0.0/16","nextHop":"10.130.0.1"},{"dest":"169.254.0.5/32","nextHop":"10.130.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.130.0.1"}],"ip_address":"10.130.0.198/23","gateway_ip":"10.130.0.1","role":"primary"}}'
k8s.v1.cni.cncf.io/network-status: |-
[{
"name": "ovn-kubernetes",
"interface": "eth0",
"ips": [
"10.130.0.198"
],
"mac": "0a:58:0a:82:00:c6",
"default": true,
"dns": {}
}]
creationTimestamp: "2025-03-03T09:18:12Z"
generateName: must-gather-
labels:
app: must-gather
name: must-gather-h7lf5
namespace: openshift-must-gather-c5jrn
resourceVersion: "167237"
uid: 24a512a2-6c2d-4060-9262-2a611dea7ee5
spec:
containers:
- command:
- /bin/bash
- -c
- "\necho \"volume percentage checker started.....\"\nwhile true; do \ndisk_usage=$(du
-s \"/must-gather\" | awk '{print $1}')\ndisk_space=$(df -P \"/must-gather\"
| awk 'NR==2 {print $2}')\nusage_percentage=$(( (disk_usage * 100) / disk_space
))\necho \"volume usage percentage $usage_percentage\" \nif [ \"$usage_percentage\"
-gt \"30\" ]; then \n\techo \"Disk usage exceeds the volume percentage of 30
for mounted directory. Exiting...\" \n\t# kill gathering process in gather container
to prevent disk to use more.\n\tpkill --signal SIGKILL -f /usr/bin/gather\n\texit
1\nfi\nsleep 5\ndone & /usr/bin/gather; sync"
env:
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
image: ghcr.io/complianceascode/must-gather-ocp:latest
imagePullPolicy: IfNotPresent
name: gather
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /must-gather
name: must-gather-output
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-xs46l
readOnly: true
- command:
- /bin/bash
- -c
- 'trap : TERM INT; sleep infinity & wait'
image: ghcr.io/complianceascode/must-gather-ocp:latest
imagePullPolicy: IfNotPresent
name: copy
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /must-gather
name: must-gather-output
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-xs46l
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
imagePullSecrets:
- name: default-dockercfg-66ntk
nodeName: ip-10-0-42-65.us-east-2.compute.internal
nodeSelector:
kubernetes.io/os: linux
node-role.kubernetes.io/master: ""
preemptionPolicy: PreemptLowerPriority
priority: 2000000000
priorityClassName: system-cluster-critical
restartPolicy: Never
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 0
tolerations:
- operator: Exists
volumes:
- emptyDir: {}
name: must-gather-output
- name: kube-api-access-xs46l
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
- configMap:
items:
- key: service-ca.crt
path: service-ca.crt
name: openshift-service-ca.crt
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2025-03-03T09:18:13Z"
status: "True"
type: PodReadyToStartContainers
- lastProbeTime: null
lastTransitionTime: "2025-03-03T09:18:12Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2025-03-03T09:18:13Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2025-03-03T09:18:13Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2025-03-03T09:18:12Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: cri-o://7eb9644334d2508d67181312112fff37af63d134b1fcd98caf9626887c4ac0d0
image: ghcr.io/complianceascode/must-gather-ocp:latest
imageID: ghcr.io/complianceascode/must-gather-ocp@sha256:349fad86163e2fdc3577d9369f95551ce176a4c5b91f59cfdb193fd5bc3f982e
lastState: {}
name: copy
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2025-03-03T09:18:13Z"
volumeMounts:
- mountPath: /must-gather
name: must-gather-output
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-xs46l
readOnly: true
recursiveReadOnly: Disabled
- containerID: cri-o://a98eb8bc7e684931fbd929566cf986cce87c4da5858efd249a1419e45ea106c6
image: ghcr.io/complianceascode/must-gather-ocp:latest
imageID: ghcr.io/complianceascode/must-gather-ocp@sha256:349fad86163e2fdc3577d9369f95551ce176a4c5b91f59cfdb193fd5bc3f982e
lastState: {}
name: gather
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2025-03-03T09:18:13Z"
volumeMounts:
- mountPath: /must-gather
name: must-gather-output
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-xs46l
readOnly: true
recursiveReadOnly: Disabled
hostIP: 10.0.42.65
hostIPs:
- ip: 10.0.42.65
phase: Running
podIP: 10.130.0.198
podIPs:
- ip: 10.130.0.198
qosClass: BestEffort
startTime: "2025-03-03T09:18:12Z"
@xiaojiey Will take a look later this week. What was the change that enabled you to reproduce the issue?
@xiaojiey Will take a look later this week. What was the change that enabled you to reproduce the issue? I executed below command to add taint for each node:
% for node inoc get node --no-headers | awk '{print $1}'; do kubectl taint nodes $node node-role.kubernetes.io/master=:NoSchedule --overwrite=true; done