node-cleanup deleter process repeatedly deleting unused, but valid PVs

[scole-scea]

What happened:

In a cluster where the node name != kubernetes.io/hostname, the checker for node deletion often (but not always) thinks new PVs belong to deleted nodes, and deletes them. (Then the provisioner reprovisions them a few seconds later, causing constant churn.) Thankfully this only happens to unused volumes.

What you expected to happen:

I expect provisioned, valid PVs to be ignored by the deleter, even if they're not used.

How to reproduce it:

In a cluster where node name != kubernetes.io/hostname, launch the storage provisioner and add the node-cleanup service. Notice churn in the volumes.

Anything else we need to know?:

Environment:

• CSI Driver version: v2.6.0
• Kubernetes version (use kubectl version): v1.24.17-eks-5e0fdde
• OS (e.g. from /etc/os-release): Amazon Linux version 2
• Kernel (e.g. uname -a): 5.10.201-191.748.amzn2.x86_64
• Install tools: helm
• Others:

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[scole66]

/remove-lifecycle rotten

[niranjandarshann]

@scole-scea I had also faced this problem but overcome up with using consistent labelling of the nodes. Ensure that the kubernetes.io/hostname label matches the name of the node by setting it on each node correctly. We can achieve this using an init container to set the label.

[niranjandarshann]

Here are the Daemonset.yaml under

spec: 
   initContainers:
      - name: set-node-label
        image: busybox
        command:
        - sh
        - -c
        - |
          NODE_NAME=$(cat /etc/hostname)
          kubectl label node $NODE_NAME kubernetes.io/hostname=$NODE_NAME --overwrite

[niranjandarshann]

Hope it may help but yes the problem is valid as it delete the valid PVs.

[scole-scea]

Unfortunately, that's not a great answer for our environment. Hostnames are set by legacy rules; k8s node names have always been different and now are expected, etc, etc.

I'd love it if someone who knew this code could make the "easy change" (if such a thing exists); else I'll get to it maybe sometime this year.

I do appreciate your comments, though. It means my guesses about what the problem is are correct, which is a lovely confirmation.

[niranjandarshann]

Thank you for your feedback. I provided the most probable solution but yes, I can understand that our environment have specific requirement. We should look into it.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.