Helm Chart installation on AWS EKS: Unable to attach or mount volumes

[DarkoDevops]

I installed DevLake using Helm chart on my AWS EKS cluster. Used previous instructions from ticket: https://github.com/apache/incubator-devlake-helm-chart/issues/87 but, without success. Still not working.

Helm values.yaml looks like:

replicaCount: 1
imageTag: v0.16.0-beta17

mysql:
  useExternal: false
  externalServer: 127.0.0.1
  externalPort: 3306
  username: merico
  password: merico
  database: lake
  rootPassword: admin
  storage:
    class: "gp2"
    size: 50Gi

  image:
    repository: mysql
    tag: 8
    pullPolicy: IfNotPresent

  resources: {}

  nodeSelector: {}
  tolerations: []

  affinity: {}


grafana:
  image:
    repository: apache/devlake-dashboard
    pullPolicy: Always

  useExternal: false

  externalUrl: ''

  resources: {}

  nodeSelector: {}

  tolerations: []

  affinity: {}


lake:
  image:
    repository: apache/devlake
    pullPolicy: Always
  storage:
    class: "gp2"
    size: 100Mi
  dotenv:
    API_TIMEOUT: "120s"
    API_RETRY: "3"
    API_REQUESTS_PER_HOUR: "10000"
    PIPELINE_MAX_PARALLEL: "1"
    IN_SECURE_SKIP_VERIFY: "false"

  hostNetwork: false

  resources: {}

  nodeSelector: {}

  tolerations: []

  affinity: {}

  loggingDir: "/app/logs"
  loggingLevel: "info"

ui:
  image:
    repository: apache/devlake-config-ui
    pullPolicy: Always

  resources: {}

  nodeSelector: {}

  tolerations: []

  affinity: {}

  basicAuth:
    enabled: false
    user: admin
    password: admin
    useSecret: false
    autoCreateSecret: true
    secretName: devlake-auth


alpine:
  image:
    repository: alpine
    tag: 3.16
    pullPolicy: IfNotPresent

service:
  type: NodePort
  uiPort: 32001
  grafanaPort : 32002

ingress:
  enabled: true
  enableHttps: false
  className: "alb"
  prefix: /
  tlsSecretName: ""
  httpPort: 80
  httpsPort: 443
  useDefaultNginx: false
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip

option:
  localtime: /etc/localtime

  database: mysql

  useConnectionDetailsSecret: false

  connectionSecretName: devlake-db-connection

  autoCreateSecret: true

I get multiple errors like:

Unable to attach or mount volumes: unmounted volumes=[devlake-lake-config], unattached volumes=[devlake-lake-localtime kube-api-access-r7dgn devlake-lake-config]: timed out waiting for the condition

Unable to attach or mount volumes: unmounted volumes=[devlake-lake-config], unattached volumes=[kube-api-access-r7dgn devlake-lake-config devlake-lake-localtime]: timed out waiting for the condition

Unable to attach or mount volumes: unmounted volumes=[devlake-lake-config], unattached volumes=[devlake-lake-config devlake-lake-localtime kube-api-access-r7dgn]: timed out waiting for the condition

AttachVolume.Attach failed for volume "pvc-09b75f09-f0e4-4700-b42c-e9f901a13bf2" : volume attachment is being deleted

MountVolume.WaitForAttach failed for volume "pvc-09b75f09-f0e4-4700-b42c-e9f901a13bf2" : volume attachment is being deleted

Also tried to set storage class to empty (default) and same behaviour. Any idea?

[klesh]

@IronCore864 @JorgeGar would you mind taking a look at this?

[IronCore864]

@DarkoDevops can you run kubectl get/describe pv/pvc and see what you can get from the output?

A wild guess: do you have the aws-ebs-csi-driver addon enabled in your EKS?

[DarkoDevops]

@IronCore864 I will run it and post it.

Yes, I have aws-ebs-csi-driver addon.

[DarkoDevops]

pv/pvc describe:

pvc-09b75f09-f0e4-4700-b42c-e9f901a13bf2   1Gi        RWO            Delete           Bound    default/devlake-lake-config-devlake-lake-0   gp2                     65d
pvc-16438fe5-4163-441b-be46-2a9d8920b99f   50Gi       RWO            Delete           Bound    default/devlake-mysql-data-devlake-mysql-0   gp2                     65d

Name:              pvc-09b75f09-f0e4-4700-b42c-e9f901a13bf2
Labels:            topology.kubernetes.io/region=eu-central-1
                   topology.kubernetes.io/zone=eu-central-1a
Annotations:       kubernetes.io/createdby: aws-ebs-dynamic-provisioner
                   pv.kubernetes.io/bound-by-controller: yes
                   pv.kubernetes.io/migrated-to: ebs.csi.aws.com
                   pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
Finalizers:        [kubernetes.io/pv-protection external-attacher/ebs-csi-aws-com]
StorageClass:      gp2
Status:            Bound
Claim:             default/devlake-lake-config-devlake-lake-0
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          1Gi
Node Affinity:     
  Required Terms:  
    Term 0:        topology.kubernetes.io/zone in [eu-central-1a]
                   topology.kubernetes.io/region in [eu-central-1]
Message:           
Source:
    Type:       AWSElasticBlockStore (a Persistent Disk resource in AWS)
    VolumeID:   aws://eu-central-1a/vol-0b203af215cc352d9
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
Events:         <none>

Name:              pvc-16438fe5-4163-441b-be46-2a9d8920b99f
Labels:            topology.kubernetes.io/region=eu-central-1
                   topology.kubernetes.io/zone=eu-central-1b
Annotations:       kubernetes.io/createdby: aws-ebs-dynamic-provisioner
                   pv.kubernetes.io/bound-by-controller: yes
                   pv.kubernetes.io/migrated-to: ebs.csi.aws.com
                   pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
Finalizers:        [kubernetes.io/pv-protection external-attacher/ebs-csi-aws-com]
StorageClass:      gp2
Status:            Bound
Claim:             default/devlake-mysql-data-devlake-mysql-0
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          50Gi
Node Affinity:     
  Required Terms:  
    Term 0:        topology.kubernetes.io/zone in [eu-central-1b]
                   topology.kubernetes.io/region in [eu-central-1]
Message:           
Source:
    Type:       AWSElasticBlockStore (a Persistent Disk resource in AWS)
    VolumeID:   aws://eu-central-1b/vol-07a37b8cc48ea301f
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
Events:         <none>

[JorgeGar]

I'm running similar values without any issues. I have mysql in an external database, but that's not the one giving you issues, as per what I see in your comment @DarkoDevops. The key difference I guess is that I have aws ebs csi driver installed in the cluster, rather than it being an add-on. Could you try and get logs from it? Also, which EKS version are you using?

[DarkoDevops]

@JorgeGar It's 1.25

It's installed, in cluster, also, added policy, not the add-on.

[JorgeGar]

@JorgeGar It's 1.25

It's installed, in cluster, also, added policy, not the add-on.

Have you checked the logs of the attacher?

[DarkoDevops]

This is what I have of the pods for ebs:

ebs-csi-controller-78bdbf98ff-574px   5/5     Running   0            2d5h
ebs-csi-controller-78bdbf98ff-hwktx   5/5     Running   2 (8d ago)   8d
ebs-csi-node-46nh7                    3/3     Running   0            8d
ebs-csi-node-dthdq                    3/3     Running   0            2d5h
ebs-csi-node-fpz98                    3/3     Running   0            8d

[JorgeGar]

Honestly, at this point, I'm following the troubleshooting page from AWS, but... On the ebs-csi-controller, you should see a container called "csi-attacher". Try getting logs from there.

[BrookeKatalon]

Seeing the same issue with imageTag: v0.17.0-beta5

containers have errors like: MountVolume.SetUp failed for volume "devlake-mysql-localtime" : hostPath type check failed: /etc/localtime is not a file