aws-efs-csi-driver Fargate statefulset pending when use volumeClaimTemplates(after restart then running)

/kind bug

What happened? Fargate Statefulset pending when use volumeClaimTemplates, the describe shows Warning FailedScheduling 40s fargate-scheduler Pod not supported on Fargate: volumes not supported: efs-storage not supported because: PVC efs-storage-efs-app-sts1-0 not bound but the pvc already created and bound persistentvolumeclaim/efs-storage-efs-app-sts1-0 Bound pvc-a4bb9d1e-c307-411f-b2e6-53b09d6f5c0f 1Gi RWO efs-sc-dyna 2m21s

Restart will make the pod running

What you expected to happen? Should run at the first time instead of restart it

How to reproduce it (as minimally and precisely as possible)?

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc-dyna
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-xxxxxxxxxx
  directoryPerms: "777"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: efs-app-sts1
spec:
  selector:
    matchLabels:
      app: test-efs
  serviceName: efs-app
  replicas: 1
  template:
    metadata:
      labels:
        app: test-efs
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: linux
        image: amazonlinux:2
        command: ["/bin/sh"]
        args: ["-c", "while true; do echo $(date -u) >> /efs-data/out.txt; sleep 5; done"]
        volumeMounts:
        - name: efs-storage
          mountPath: /efs-data
  volumeClaimTemplates:
  - metadata:
      name: efs-storage
    spec:
      accessModes: [ReadWriteOnce]
      storageClassName: efs-sc-dyna
      resources:
        requests:
          storage: 1Gi

Anything else we need to know?:

Environment

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:14:10Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"darwin/amd64"} Kustomize Version: v4.5.4 Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.10-eks-15b7512", GitCommit:"cd6399691d9b1fed9ec20c9c5e82f5993c3f42cb", GitTreeState:"clean", BuildDate:"2022-08-31T19:17:01Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}
Driver version: NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION aws-efs-csi-driver kube-system 10 2022-10-15 21:02:06.538321 +0800 +08 deployed aws-efs-csi-driver-2.2.9 1.4.2

Oct 15 '22 13:10 debu99

I am experiencing exactly this problem. I've tried on Fargate EKS 1.23 and 1.24 and see the same issue.

Dec 15 '22 00:12 charlesakalugwu

Same issue here

Jan 17 '23 14:01 ahlae

fargate does not support dynamic pv. you must create pv by manual

Feb 12 '23 12:02 fyqtian

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

May 13 '23 12:05 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

Jun 12 '23 13:06 k8s-triage-robot

/remove-lifecycle rotten

Same issue for me.

I am using the following workaround with ArgoCD automation, using PersistentVolumeClaim with argocd.argoproj.io/sync-wave annotation. It works fine after many re-creations.

Before:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  labels:
    app.kubernetes.io/name: mysql
    app.kubernetes.io/instance: my-app
spec:
  serviceName: mysql
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: mysql
        app.kubernetes.io/instance: my-app
        node: fargate
    spec:
      containers:
        - name: mysql
          image: "mysql:x.x"
          imagePullPolicy: Always
          # irrelevant configs omitted
          volumeMounts:
            - name: mysql-data
              mountPath: /var/lib/mysql
  volumeClaimTemplates:
    - metadata:
        name: mysql-data
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: efs-sc
        resources:
          requests:
            storage: 1Gi

After:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  labels:
    app.kubernetes.io/name: mysql
    app.kubernetes.io/instance: my-app
spec:
  serviceName: mysql
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: mysql
        app.kubernetes.io/instance: my-app
        node: fargate
    spec:
      containers:
        - name: mysql
          image: "mysql:x.x"
          imagePullPolicy: Always
          # irrelevant configs omitted
          volumeMounts:
            - name: mysql-data
              mountPath: /var/lib/mysql
      volumes:
        - name: mysql-data
          persistentVolumeClaim:
            claimName: mysql-data-claim

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-data-claim
  annotations:
    argocd.argoproj.io/sync-wave: "-5"
spec:
  accessModes: [ "ReadWriteOnce" ]
  storageClassName: efs-sc
  resources:
    requests:
      storage: 1Gi

But I hope I can use volumeClaimTemplates and I don't need argocd.argoproj.io/sync-wave: "-5" for this.

Jun 13 '23 02:06 gecko655

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

Jan 22 '24 10:01 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

Feb 21 '24 10:02 k8s-triage-robot

#'|,–n-,IK silk|(`’)|Tata ex২⟩\konabari, gazipur(..):|-|'-'íì>⟨⟩<{)

Mar 02 '24 06:03 Mohammad-gif

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Apr 01 '24 07:04 k8s-triage-robot

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen

Mark this issue as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Apr 01 '24 07:04 k8s-ci-robot

aws-efs-csi-driver aws-efs-csi-driver copied to clipboard

Fargate statefulset pending when use volumeClaimTemplates(after restart then running)

aws-efs-csi-driver
aws-efs-csi-driver copied to clipboard