Not able to mount volume: Please set the "region" parameter in the efs-utils configuration file

[TBeijen]

/kind bug

What happened?

A PVC provisioned by aws-efs-csi-driver, previously mounting fine, fails to mount after pod restart due to an error seemingly originating from efs-utils:

Error retrieving region. Please set the "region" parameter in the efs-utils configuration file.

What you expected to happen?

PVC mounting fine

How to reproduce it (as minimally and precisely as possible)?

Not (yet) clear since all other PVCs using same storageclass, on same and identical clusters so far run fine. Error seems specific for the volume since on replacing pod it persists (PVC used by single statefulset pod).

Anything else we need to know?:

Environment

• Kubernetes version (use kubectl version): v1.22.9-eks-a64ea69
• Driver version: v1.4.0

Recently upgraded efs-csi-driver from 1.3.5 to 1.4.0. Affected PVC has been created by 1.4.0 but so have others that work fine.

Errors

# describe pod that needs the volume
 Normal   Scheduled    93s   default-scheduler  Successfully assigned sentry/sentry-kafka-2 to ip-100-64-61-170.eu-west-1.compute.internal
  Warning  FailedMount  93s   kubelet            MountVolume.SetUp failed for volume "pvc-d0c863ef-419a-4975-9b2d-a4f3b39b3f66" : rpc error: code = Internal desc = Could not mount "fs-0bb74173d3eec42e2:/" at "/var/lib/kubelet/pods/f81eaa1a-36cf-4ee4-ad3d-8b89ef40081b/volumes/kubernetes.io~csi/pvc-d0c863ef-419a-4975-9b2d-a4f3b39b3f66/mount": mount failed: exit status 1
Mounting command: mount
Mounting arguments: -t efs -o accesspoint=fsap-04ef996767a44a910,tls fs-0bb74173d3eec42e2:/ /var/lib/kubelet/pods/f81eaa1a-36cf-4ee4-ad3d-8b89ef40081b/volumes/kubernetes.io~csi/pvc-d0c863ef-419a-4975-9b2d-a4f3b39b3f66/mount
Output: Error retrieving region. Please set the "region" parameter in the efs-utils configuration file.

# efs-csi-node
efs-csi-node-577kt efs-plugin Mounting arguments: -t efs -o accesspoint=fsap-04ef996767a44a910,tls fs-0bb74173d3eec42e2:/ /var/lib/kubelet/pods/f81eaa1a-36cf-4ee4-ad3d-8b89ef40081b/volumes/kubernetes.io~csi/pvc-d0c863ef-419a-4975-9b2d-a4f3b39b3f66/mount
efs-csi-node-577kt efs-plugin Output: Error retrieving region. Please set the "region" parameter in the efs-utils configuration file.
efs-csi-node-577kt efs-plugin W0726 04:40:24.348078       1 node.go:158] Use of 'tls' under mountOptions is deprecated with this driver since tls is enabled by default. To disable it, set encrypt in transit in the volumeContext, e.g. 'encryptInTransit: true'

Storageclass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    meta.helm.sh/release-name: aws-efs-csi-driver
    meta.helm.sh/release-namespace: kube-system
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2022-04-13T14:48:13Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: efs
  resourceVersion: "77747"
  uid: 264ae9fd-6f04-4e86-b2f0-092b38c16ebb
mountOptions:
- tls
parameters:
  basePath: /dynamic_provisioning
  directoryPerms: "700"
  fileSystemId: fs-0bb74173d3eec42e2
  gidRangeEnd: "50000"
  gidRangeStart: "40000"
  provisioningMode: efs-ap
provisioner: efs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

[TBeijen]

Deleting the specific efs-csi-node pod, then after it got replaced killing the statefulset pod that tried to mount the PVC, seems to have 'fixed' things.

Not knowing the ins and outs of hwo the efs-csi-node interact, it kinda looks like the pod got into a bad state, losing it's ability (or stored state) that let's it determine the region.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[njuettner]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.