kubeblocks
kubeblocks copied to clipboard
Published
20 hours ago •
apecloud
apecloud
[BUG] delete cluster pod always Terminating with csi-hostpath-driver on k3s/kind/minikube
Describe the bug
Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:04:48.280621419 +0000 UTC m=+4797.843662028 (durationBeforeRetry 1m4s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
kbcli version
Kubernetes: v1.30.4+k3s1
KubeBlocks: 0.9.1-beta.31
kbcli: 0.9.1-beta.14
To Reproduce Steps to reproduce the behavior:
- install kubeblocks
- enable csi-hostpath-driver
kbcli addon enable csi-hostpath-driver
- create cluster
kbcli cluster create kafka kafka-dukzlp --mode='combined' --cpu=0.5 --memory=0.5 --storage=1 --availability-policy=none --termination-policy=WipeOut --version=kafka-3.3.2 --storage-enable=true --meta-storage=1 --replicas=1
kubectl get pod
NAME READY STATUS RESTARTS AGE
kafka-dukzlp-kafka-combine-0 2/2 Running 0 46s
kafka-dukzlp-kafka-exporter-0 1/1 Running 0 12s
➜ ~ kubectl get cluster
NAME CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS AGE
kafka-dukzlp kafka kafka-3.3.2 WipeOut Running 49s
- delete cluster
kbcli cluster delete kafka-dukzlp --auto-approve
- See error
kubectl get pod
NAME READY STATUS RESTARTS AGE
kafka-dukzlp-kafka-combine-0 0/2 Terminating 0 3m42s
describe its
describe its kafka-dukzlp-kafka-combine
Name: kafka-dukzlp-kafka-combine
Namespace: default
Labels: app.kubernetes.io/component=kafka-combine
app.kubernetes.io/instance=kafka-dukzlp
app.kubernetes.io/managed-by=kubeblocks
app.kubernetes.io/name=kafka-combine
apps.kubeblocks.io/cluster-uid=411c0cc0-e2f0-4735-92e5-63ecfecc4234
apps.kubeblocks.io/component-name=kafka-combine
clusterdefinition.kubeblocks.io/name=kafka
clusterversion.kubeblocks.io/name=kafka-3.3.2
componentdefinition.kubeblocks.io/name=kafka-combine
helm.sh/chart=kafka-cluster-0.9.0
Annotations: app.kubernetes.io/component: kafka-combine
apps.kubeblocks.io/service-version: 3.3.2
config.kubeblocks.io/tpl-kafka-configuration-tpl: kafka-dukzlp-kafka-combine-kafka-configuration-tpl
config.kubeblocks.io/tpl-kafka-jmx-configuration-tpl: kafka-dukzlp-kafka-combine-kafka-jmx-configuration-tpl
config.kubeblocks.io/tpl-kafka-scripts-tools-tpl: kafka-dukzlp-kafka-combine-kafka-scripts-tools-tpl
config.kubeblocks.io/tpl-kafka-scripts-tpl: kafka-dukzlp-kafka-combine-kafka-scripts-tpl
kubeblocks.io/extra-env:
{"KB_KAFKA_ENABLE_SASL":"false","KB_KAFKA_BROKER_HEAP":"-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64","KB_KAFKA_CONTROLLER_HEAP":...
kubeblocks.io/generation: 3
monitor.kubeblocks.io/path.headless.its: /metrics
monitor.kubeblocks.io/port.headless.its: 5556
monitor.kubeblocks.io/scheme.headless.its: http
monitor.kubeblocks.io/scrape.headless.its: true
API Version: workloads.kubeblocks.io/v1alpha1
Kind: InstanceSet
Metadata:
Creation Timestamp: 2024-09-30T03:00:37Z
Deletion Grace Period Seconds: 0
Deletion Timestamp: 2024-09-30T03:02:38Z
Finalizers:
instanceset.workloads.kubeblocks.io/finalizer
Generation: 6
Owner References:
API Version: apps.kubeblocks.io/v1alpha1
Block Owner Deletion: true
Controller: true
Kind: Component
Name: kafka-dukzlp-kafka-combine
UID: 1a9d3526-66f9-4f26-b773-8f63e67758c4
Resource Version: 14385
UID: 5c6a4d9a-0a2e-4a20-b88c-6f606b3c6517
Spec:
Default Template Ordinals:
Member Update Strategy: BestEffortParallel
Min Ready Seconds: 0
Parallel Pod Management Concurrency: 100%
Pod Management Policy: Parallel
Pod Update Policy: PreferInPlace
Replicas: 1
Selector:
Match Labels:
app.kubernetes.io/instance: kafka-dukzlp
app.kubernetes.io/managed-by: kubeblocks
app.kubernetes.io/name: kafka-combine
apps.kubeblocks.io/component-name: kafka-combine
Template:
Metadata:
Annotations:
apps.kubeblocks.io/component-replicas: 1
Labels:
app.kubernetes.io/component: kafka-combine
app.kubernetes.io/instance: kafka-dukzlp
app.kubernetes.io/managed-by: kubeblocks
app.kubernetes.io/name: kafka-combine
app.kubernetes.io/version: kafka-combine
apps.kubeblocks.io/cluster-uid: 411c0cc0-e2f0-4735-92e5-63ecfecc4234
apps.kubeblocks.io/component-name: kafka-combine
clusterdefinition.kubeblocks.io/name: kafka
clusterversion.kubeblocks.io/name: kafka-3.3.2
componentdefinition.kubeblocks.io/name: kafka-combine
helm.sh/chart: kafka-cluster-0.9.0
Spec:
Affinity:
Node Affinity:
Preferred During Scheduling Ignored During Execution:
Preference:
Match Expressions:
Key: kb-data
Operator: In
Values:
true
Weight: 100
Pod Anti Affinity:
Containers:
Command:
/scripts/kafka-server-setup.sh
Env:
Name: SUPER_USER
Value From:
Secret Key Ref:
Key: username
Name: kafka-dukzlp-kafka-combine-account-admin
Name: KB_POD_NAME
Value From:
Field Ref:
API Version: v1
Field Path: metadata.name
Name: KB_POD_UID
Value From:
Field Ref:
API Version: v1
Field Path: metadata.uid
Name: KB_NAMESPACE
Value From:
Field Ref:
API Version: v1
Field Path: metadata.namespace
Name: KB_SA_NAME
Value From:
Field Ref:
API Version: v1
Field Path: spec.serviceAccountName
Name: KB_NODENAME
Value From:
Field Ref:
API Version: v1
Field Path: spec.nodeName
Name: KB_HOST_IP
Value From:
Field Ref:
API Version: v1
Field Path: status.hostIP
Name: KB_POD_IP
Value From:
Field Ref:
API Version: v1
Field Path: status.podIP
Name: KB_POD_IPS
Value From:
Field Ref:
API Version: v1
Field Path: status.podIPs
Name: KB_HOSTIP
Value From:
Field Ref:
API Version: v1
Field Path: status.hostIP
Name: KB_PODIP
Value From:
Field Ref:
API Version: v1
Field Path: status.podIP
Name: KB_PODIPS
Value From:
Field Ref:
API Version: v1
Field Path: status.podIPs
Name: KB_POD_FQDN
Value: $(KB_POD_NAME).kafka-dukzlp-kafka-combine-headless.$(KB_NAMESPACE).svc
Name: BITNAMI_DEBUG
Value: true
Name: MY_POD_IP
Value: $(KB_PODIP)
Name: MY_POD_NAME
Value: $(KB_POD_NAME)
Name: KAFKA_ENABLE_KRAFT
Value: yes
Name: KAFKA_CFG_PROCESS_ROLES
Value: broker,controller
Name: KAFKA_CFG_CONTROLLER_LISTENER_NAMES
Value: CONTROLLER
Name: KAFKA_CFG_INTER_BROKER_LISTENER_NAME
Value: INTERNAL
Name: KAFKA_CFG_LISTENERS
Value: CONTROLLER://:9093,INTERNAL://:9094,CLIENT://:9092
Name: KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP
Value: CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,CLIENT:PLAINTEXT
Name: KAFKA_CFG_ADVERTISED_LISTENERS
Value: INTERNAL://$(KB_POD_IP):9094,CLIENT://$(KB_POD_IP):9092
Name: KAFKA_CFG_INITIAL_BROKER_REGISTRATION_TIMEOUT_MS
Value: 240000
Name: ALLOW_PLAINTEXT_LISTENER
Value: yes
Name: JMX_PORT
Value: 5555
Name: KAFKA_VOLUME_DIR
Value: /bitnami/kafka
Name: KAFKA_CFG_METADATA_LOG_DIR
Value: /bitnami/kafka/metadata
Name: KAFKA_LOG_DIR
Value: /bitnami/kafka/data
Name: KAFKA_HEAP_OPTS
Value: -XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64
Name: SERVER_PROP_FILE
Value: /scripts/server.properties
Name: KAFKA_KRAFT_CLUSTER_ID
Value: $(KB_CLUSTER_UID)
Name: KAFKA_CFG_SUPER_USERS
Value: User:$(SUPER_USER)
Name: KB_KAFKA_SASL_CONFIG_PATH
Value: /tools/server-jaas.properties
Env From:
Config Map Ref:
Name: kafka-dukzlp-kafka-combine-env
Optional: false
Image: docker.io/apecloud/kafka:3.3.2-debian-11-r54
Image Pull Policy: IfNotPresent
Liveness Probe:
Failure Threshold: 3
Initial Delay Seconds: 10
Period Seconds: 10
Success Threshold: 1
Tcp Socket:
Port: kafka-ctrlr
Timeout Seconds: 5
Name: kafka
Ports:
Container Port: 9092
Name: kafka-client
Protocol: TCP
Container Port: 9093
Name: kafka-ctrlr
Protocol: TCP
Container Port: 9094
Name: kafka-internal
Protocol: TCP
Resources:
Limits:
Cpu: 500m
Memory: 512Mi
Requests:
Cpu: 500m
Memory: 512Mi
Security Context:
Allow Privilege Escalation: false
Run As Non Root: true
Run As User: 1001
Startup Probe:
Failure Threshold: 30
Initial Delay Seconds: 5
Period Seconds: 10
Success Threshold: 1
Tcp Socket:
Port: kafka-ctrlr
Timeout Seconds: 5
Volume Mounts:
Mount Path: /bitnami/kafka
Name: data
Mount Path: /bitnami/kafka/metadata
Name: metadata
Mount Path: /scripts/kafka-server-setup.sh
Name: scripts
Sub Path: kafka-server-setup.sh
Mount Path: /scripts/server.properties
Name: kafka-config
Sub Path: server.properties
Mount Path: /tools/client-ssl.properties
Name: tools
Sub Path: client-ssl.properties
Mount Path: /tools/server-jaas.properties
Name: tools
Sub Path: server-jaas.properties
Args:
-XX:MaxRAMPercentage=100
-XshowSettings:vm
-jar
jmx_prometheus_httpserver.jar
5556
/etc/jmx-kafka/jmx-kafka-prometheus.yml
Command:
java
Env:
Name: SUPER_USER
Value From:
Secret Key Ref:
Key: username
Name: kafka-dukzlp-kafka-combine-account-admin
Name: KB_POD_NAME
Value From:
Field Ref:
API Version: v1
Field Path: metadata.name
Name: KB_POD_UID
Value From:
Field Ref:
API Version: v1
Field Path: metadata.uid
Name: KB_NAMESPACE
Value From:
Field Ref:
API Version: v1
Field Path: metadata.namespace
Name: KB_SA_NAME
Value From:
Field Ref:
API Version: v1
Field Path: spec.serviceAccountName
Name: KB_NODENAME
Value From:
Field Ref:
API Version: v1
Field Path: spec.nodeName
Name: KB_HOST_IP
Value From:
Field Ref:
API Version: v1
Field Path: status.hostIP
Name: KB_POD_IP
Value From:
Field Ref:
API Version: v1
Field Path: status.podIP
Name: KB_POD_IPS
Value From:
Field Ref:
API Version: v1
Field Path: status.podIPs
Name: KB_HOSTIP
Value From:
Field Ref:
API Version: v1
Field Path: status.hostIP
Name: KB_PODIP
Value From:
Field Ref:
API Version: v1
Field Path: status.podIP
Name: KB_PODIPS
Value From:
Field Ref:
API Version: v1
Field Path: status.podIPs
Name: KB_POD_FQDN
Value: $(KB_POD_NAME).kafka-dukzlp-kafka-combine-headless.$(KB_NAMESPACE).svc
Name: SERVICE_PORT
Value: 5556
Env From:
Config Map Ref:
Name: kafka-dukzlp-kafka-combine-env
Optional: false
Image: docker.io/apecloud/jmx-exporter:0.18.0-debian-11-r20
Image Pull Policy: IfNotPresent
Name: jmx-exporter
Ports:
Container Port: 5556
Name: metrics
Protocol: TCP
Resources:
Limits:
Cpu: 0
Memory: 0
Security Context:
Run As Non Root: true
Run As User: 1001
Volume Mounts:
Mount Path: /etc/jmx-kafka
Name: jmx-config
Security Context:
Fs Group: 1001
Tolerations:
Effect: NoSchedule
Key: kb-data
Operator: Equal
Value: true
Volumes:
Config Map:
Default Mode: 292
Name: kafka-dukzlp-kafka-combine-kafka-configuration-tpl
Name: kafka-config
Config Map:
Default Mode: 292
Name: kafka-dukzlp-kafka-combine-kafka-jmx-configuration-tpl
Name: jmx-config
Config Map:
Default Mode: 493
Name: kafka-dukzlp-kafka-combine-kafka-scripts-tools-tpl
Name: tools
Config Map:
Default Mode: 493
Name: kafka-dukzlp-kafka-combine-kafka-scripts-tpl
Name: scripts
Empty Dir:
Name: data
Empty Dir:
Name: metadata
Update Strategy:
Member Update Strategy: BestEffortParallel
Volume Claim Templates:
Metadata:
Labels:
apps.kubeblocks.io/vct-name: data
Name: data
Spec:
Access Modes:
ReadWriteOnce
Resources:
Requests:
Storage: 1Gi
Status:
Metadata:
Labels:
apps.kubeblocks.io/vct-name: metadata
Name: metadata
Spec:
Access Modes:
ReadWriteOnce
Resources:
Requests:
Storage: 1Gi
Status:
Status:
Available Replicas: 1
Conditions:
Last Transition Time: 2024-09-30T03:01:10Z
Message:
Observed Generation: 4
Reason: Ready
Status: True
Type: InstanceReady
Last Transition Time: 2024-09-30T03:01:10Z
Message:
Observed Generation: 4
Reason: Available
Status: True
Type: InstanceAvailable
Current Replicas: 1
Current Revision: 7cc7f549b6
Current Revisions:
kafka-dukzlp-kafka-combine-0: 7cc7f549b6
Init Replicas: 0
Observed Generation: 6
Ready Replicas: 1
Replicas: 1
Update Revision: 7cc7f549b6
Update Revisions:
kafka-dukzlp-kafka-combine-0: 7cc7f549b6
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulUpdate 49m instance-set-controller update InstanceSet kafka-dukzlp-kafka-combine in InstanceSet kafka-dukzlp-kafka-combine successful
Normal SuccessfulCreate 49m instance-set-controller create ConfigMap kafka-dukzlp-kafka-combine-rsm-env in InstanceSet kafka-dukzlp-kafka-combine successful
Normal SuccessfulCreate 49m instance-set-controller create Service kafka-dukzlp-kafka-combine-headless in InstanceSet kafka-dukzlp-kafka-combine successful
Normal SuccessfulCreate 49m instance-set-controller create PersistentVolumeClaim metadata-kafka-dukzlp-kafka-combine-0 in InstanceSet kafka-dukzlp-kafka-combine successful
Normal SuccessfulCreate 49m instance-set-controller create PersistentVolumeClaim data-kafka-dukzlp-kafka-combine-0 in InstanceSet kafka-dukzlp-kafka-combine successful
Normal SuccessfulCreate 49m instance-set-controller create Pod kafka-dukzlp-kafka-combine-0 in InstanceSet kafka-dukzlp-kafka-combine successful
Normal SuccessfulDelete 47m instance-set-controller delete ConfigMap kafka-dukzlp-kafka-combine-rsm-env in InstanceSet kafka-dukzlp-kafka-combine successful
Normal SuccessfulDelete 47m instance-set-controller delete Service kafka-dukzlp-kafka-combine-headless in InstanceSet kafka-dukzlp-kafka-combine successful
Normal SuccessfulDelete 47m instance-set-controller delete Pod kafka-dukzlp-kafka-combine-0 in InstanceSet kafka-dukzlp-kafka-combine successful
k3s logs
I0930 03:02:37.766359 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/a1c54c0d-dfa5-47b8-b399-62de6182fac4-scripts\") pod \"a1c54c0d-dfa5-47b8-b399-62de6182fac4\" (UID: \"a1c54c0d-dfa5-47b8-b399-62de6182fac4\") "
I0930 03:02:37.767367 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c69wh\" (UniqueName: \"kubernetes.io/projected/a1c54c0d-dfa5-47b8-b399-62de6182fac4-kube-api-access-c69wh\") pod \"a1c54c0d-dfa5-47b8-b399-62de6182fac4\" (UID: \"a1c54c0d-dfa5-47b8-b399-62de6182fac4\") "
I0930 03:02:37.775634 87 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c54c0d-dfa5-47b8-b399-62de6182fac4-kube-api-access-c69wh" (OuterVolumeSpecName: "kube-api-access-c69wh") pod "a1c54c0d-dfa5-47b8-b399-62de6182fac4" (UID: "a1c54c0d-dfa5-47b8-b399-62de6182fac4"). InnerVolumeSpecName "kube-api-access-c69wh". PluginName "kubernetes.io/projected", VolumeGidValue ""
I0930 03:02:37.777571 87 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c54c0d-dfa5-47b8-b399-62de6182fac4-scripts" (OuterVolumeSpecName: "scripts") pod "a1c54c0d-dfa5-47b8-b399-62de6182fac4" (UID: "a1c54c0d-dfa5-47b8-b399-62de6182fac4"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/configmap", VolumeGidValue ""
I0930 03:02:37.868098 87 reconciler_common.go:289] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/a1c54c0d-dfa5-47b8-b399-62de6182fac4-scripts\") on node \"k3d-default-server-0\" DevicePath \"\""
I0930 03:02:37.868127 87 reconciler_common.go:289] "Volume detached for volume \"kube-api-access-c69wh\" (UniqueName: \"kubernetes.io/projected/a1c54c0d-dfa5-47b8-b399-62de6182fac4-kube-api-access-c69wh\") on node \"k3d-default-server-0\" DevicePath \"\""
I0930 03:02:38.097577 87 scope.go:117] "RemoveContainer" containerID="dcac400ea69c6958d0ff23937a6afc219c4ce1da53580dcb78ff325181108e6f"
I0930 03:02:38.110470 87 scope.go:117] "RemoveContainer" containerID="dcac400ea69c6958d0ff23937a6afc219c4ce1da53580dcb78ff325181108e6f"
E0930 03:02:38.111458 87 remote_runtime.go:432] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"dcac400ea69c6958d0ff23937a6afc219c4ce1da53580dcb78ff325181108e6f\": not found" containerID="dcac400ea69c6958d0ff23937a6afc219c4ce1da53580dcb78ff325181108e6f"
I0930 03:02:38.111535 87 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"containerd","ID":"dcac400ea69c6958d0ff23937a6afc219c4ce1da53580dcb78ff325181108e6f"} err="failed to get container status \"dcac400ea69c6958d0ff23937a6afc219c4ce1da53580dcb78ff325181108e6f\": rpc error: code = NotFound desc = an error occurred when try to find container \"dcac400ea69c6958d0ff23937a6afc219c4ce1da53580dcb78ff325181108e6f\": not found"
I0930 03:02:38.904674 87 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1c54c0d-dfa5-47b8-b399-62de6182fac4" path="/var/lib/kubelet/pods/a1c54c0d-dfa5-47b8-b399-62de6182fac4/volumes"
I0930 03:02:40.490442 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:40.490721 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-scripts\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:40.491017 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:40.491066 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"tools\" (UniqueName: \"kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-tools\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:40.491098 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"jmx-config\" (UniqueName: \"kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-jmx-config\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:40.492822 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"kafka-config\" (UniqueName: \"kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-kafka-config\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:40.492980 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t8r2\" (UniqueName: \"kubernetes.io/projected/33229d89-4052-46f1-9b49-ff39382a4c54-kube-api-access-6t8r2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:40.494079 87 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-jmx-config" (OuterVolumeSpecName: "jmx-config") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54"). InnerVolumeSpecName "jmx-config". PluginName "kubernetes.io/configmap", VolumeGidValue ""
I0930 03:02:40.495878 87 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-scripts" (OuterVolumeSpecName: "scripts") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54"). InnerVolumeSpecName "scripts". PluginName "kubernetes.io/configmap", VolumeGidValue ""
I0930 03:02:40.499036 87 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-kafka-config" (OuterVolumeSpecName: "kafka-config") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54"). InnerVolumeSpecName "kafka-config". PluginName "kubernetes.io/configmap", VolumeGidValue ""
E0930 03:02:40.500476 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:41.000438084 +0000 UTC m=+4670.562037248 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:02:40.500641 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:41.0005735 +0000 UTC m=+4670.562172540 (durationBeforeRetry 500ms). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
I0930 03:02:40.500786 87 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33229d89-4052-46f1-9b49-ff39382a4c54-kube-api-access-6t8r2" (OuterVolumeSpecName: "kube-api-access-6t8r2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54"). InnerVolumeSpecName "kube-api-access-6t8r2". PluginName "kubernetes.io/projected", VolumeGidValue ""
I0930 03:02:40.503795 87 operation_generator.go:887] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-tools" (OuterVolumeSpecName: "tools") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54"). InnerVolumeSpecName "tools". PluginName "kubernetes.io/configmap", VolumeGidValue ""
I0930 03:02:40.594361 87 reconciler_common.go:289] "Volume detached for volume \"tools\" (UniqueName: \"kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-tools\") on node \"k3d-default-server-0\" DevicePath \"\""
I0930 03:02:40.594393 87 reconciler_common.go:289] "Volume detached for volume \"jmx-config\" (UniqueName: \"kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-jmx-config\") on node \"k3d-default-server-0\" DevicePath \"\""
I0930 03:02:40.594400 87 reconciler_common.go:289] "Volume detached for volume \"kafka-config\" (UniqueName: \"kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-kafka-config\") on node \"k3d-default-server-0\" DevicePath \"\""
I0930 03:02:40.594425 87 reconciler_common.go:289] "Volume detached for volume \"kube-api-access-6t8r2\" (UniqueName: \"kubernetes.io/projected/33229d89-4052-46f1-9b49-ff39382a4c54-kube-api-access-6t8r2\") on node \"k3d-default-server-0\" DevicePath \"\""
I0930 03:02:40.594450 87 reconciler_common.go:289] "Volume detached for volume \"scripts\" (UniqueName: \"kubernetes.io/configmap/33229d89-4052-46f1-9b49-ff39382a4c54-scripts\") on node \"k3d-default-server-0\" DevicePath \"\""
I0930 03:02:41.003851 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:41.004137 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
E0930 03:02:41.008360 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:42.008326459 +0000 UTC m=+4671.569925498 (durationBeforeRetry 1s). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:02:41.008560 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:42.008534417 +0000 UTC m=+4671.570133498 (durationBeforeRetry 1s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
I0930 03:02:41.118935 87 scope.go:117] "RemoveContainer" containerID="275c0f87587ca4a9ebb26b1f62d84bfece6d18ae37b091895bb991d04da133e4"
I0930 03:02:41.131600 87 scope.go:117] "RemoveContainer" containerID="1e585e05b98c1cdfa07afa8092cca91815d851fd8967009c9d444394968933b8"
I0930 03:02:41.142411 87 scope.go:117] "RemoveContainer" containerID="275c0f87587ca4a9ebb26b1f62d84bfece6d18ae37b091895bb991d04da133e4"
E0930 03:02:41.142974 87 remote_runtime.go:432] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"275c0f87587ca4a9ebb26b1f62d84bfece6d18ae37b091895bb991d04da133e4\": not found" containerID="275c0f87587ca4a9ebb26b1f62d84bfece6d18ae37b091895bb991d04da133e4"
I0930 03:02:41.143089 87 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"containerd","ID":"275c0f87587ca4a9ebb26b1f62d84bfece6d18ae37b091895bb991d04da133e4"} err="failed to get container status \"275c0f87587ca4a9ebb26b1f62d84bfece6d18ae37b091895bb991d04da133e4\": rpc error: code = NotFound desc = an error occurred when try to find container \"275c0f87587ca4a9ebb26b1f62d84bfece6d18ae37b091895bb991d04da133e4\": not found"
I0930 03:02:41.143289 87 scope.go:117] "RemoveContainer" containerID="1e585e05b98c1cdfa07afa8092cca91815d851fd8967009c9d444394968933b8"
E0930 03:02:41.144119 87 remote_runtime.go:432] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"1e585e05b98c1cdfa07afa8092cca91815d851fd8967009c9d444394968933b8\": not found" containerID="1e585e05b98c1cdfa07afa8092cca91815d851fd8967009c9d444394968933b8"
I0930 03:02:41.144273 87 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"containerd","ID":"1e585e05b98c1cdfa07afa8092cca91815d851fd8967009c9d444394968933b8"} err="failed to get container status \"1e585e05b98c1cdfa07afa8092cca91815d851fd8967009c9d444394968933b8\": rpc error: code = NotFound desc = an error occurred when try to find container \"1e585e05b98c1cdfa07afa8092cca91815d851fd8967009c9d444394968933b8\": not found"
I0930 03:02:42.012815 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:42.013091 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
E0930 03:02:42.017587 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:44.017545543 +0000 UTC m=+4673.579144499 (durationBeforeRetry 2s). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:02:42.018093 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:44.018078084 +0000 UTC m=+4673.579677082 (durationBeforeRetry 2s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
I0930 03:02:44.032389 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:44.032512 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
E0930 03:02:44.034265 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:48.034252377 +0000 UTC m=+4677.595851458 (durationBeforeRetry 4s). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:02:44.034879 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:48.034855169 +0000 UTC m=+4677.596454250 (durationBeforeRetry 4s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
I0930 03:02:48.071111 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:48.071309 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
E0930 03:02:48.073392 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:56.073379546 +0000 UTC m=+4685.634978502 (durationBeforeRetry 8s). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:02:48.073464 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:02:56.073454296 +0000 UTC m=+4685.635053293 (durationBeforeRetry 8s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
I0930 03:02:56.146586 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:02:56.146823 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
E0930 03:02:56.148503 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:03:12.148481508 +0000 UTC m=+4701.710080464 (durationBeforeRetry 16s). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:02:56.148622 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:03:12.148611133 +0000 UTC m=+4701.710210131 (durationBeforeRetry 16s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
I0930 03:02:58.273902 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^8eea8c5e-7ecf-11ef-8deb-8e630c9e01e2\") pod \"0745d15a-c04f-490f-b3e1-ef9e73d9eeac\" (UID: \"0745d15a-c04f-490f-b3e1-ef9e73d9eeac\") "
I0930 03:02:58.273975 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^8ee9fb8a-7ecf-11ef-8deb-8e630c9e01e2\") pod \"0745d15a-c04f-490f-b3e1-ef9e73d9eeac\" (UID: \"0745d15a-c04f-490f-b3e1-ef9e73d9eeac\") "
E0930 03:02:58.276804 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^8ee9fb8a-7ecf-11ef-8deb-8e630c9e01e2 podName:0745d15a-c04f-490f-b3e1-ef9e73d9eeac nodeName:}" failed. No retries permitted until 2024-09-30 03:05:00.276780509 +0000 UTC m=+4809.838379465 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^8ee9fb8a-7ecf-11ef-8deb-8e630c9e01e2") pod "0745d15a-c04f-490f-b3e1-ef9e73d9eeac" (UID: "0745d15a-c04f-490f-b3e1-ef9e73d9eeac") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 8ee9fb8a-7ecf-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:02:58.276854 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^8eea8c5e-7ecf-11ef-8deb-8e630c9e01e2 podName:0745d15a-c04f-490f-b3e1-ef9e73d9eeac nodeName:}" failed. No retries permitted until 2024-09-30 03:05:00.276835426 +0000 UTC m=+4809.838434423 (durationBeforeRetry 2m2s). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^8eea8c5e-7ecf-11ef-8deb-8e630c9e01e2") pod "0745d15a-c04f-490f-b3e1-ef9e73d9eeac" (UID: "0745d15a-c04f-490f-b3e1-ef9e73d9eeac") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 8eea8c5e-7ecf-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
I0930 03:03:12.210663 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:03:12.210849 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
E0930 03:03:12.218308 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:03:44.218026668 +0000 UTC m=+4733.779737930 (durationBeforeRetry 32s). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:03:12.218400 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:03:44.218380418 +0000 UTC m=+4733.780091638 (durationBeforeRetry 32s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
I0930 03:03:44.277829 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
I0930 03:03:44.277996 87 reconciler_common.go:161] "operationExecutor.UnmountVolume started for volume \"metadata\" (UniqueName: \"kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2\") pod \"33229d89-4052-46f1-9b49-ff39382a4c54\" (UID: \"33229d89-4052-46f1-9b49-ff39382a4c54\") "
E0930 03:03:44.280565 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:04:48.280532377 +0000 UTC m=+4797.843572944 (durationBeforeRetry 1m4s). Error: UnmountVolume.TearDown failed for volume "metadata" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7abecc-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7abecc-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
E0930 03:03:44.280629 87 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 podName:33229d89-4052-46f1-9b49-ff39382a4c54 nodeName:}" failed. No retries permitted until 2024-09-30 03:04:48.280621419 +0000 UTC m=+4797.843662028 (durationBeforeRetry 1m4s). Error: UnmountVolume.TearDown failed for volume "data" (UniqueName: "kubernetes.io/csi/hostpath.csi.k8s.io^2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2") pod "33229d89-4052-46f1-9b49-ff39382a4c54" (UID: "33229d89-4052-46f1-9b49-ff39382a4c54") : kubernetes.io/csi: Unmounter.TearDownAt failed: rpc error: code = NotFound desc = volume id 2b7b49d2-7ed8-11ef-8deb-8e630c9e01e2 does not exist in the volumes list
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
Additional context Add any other context about the problem here.
