Grafana Kubernetes Pod stuck in Container Init

[wnds]

Release : 1.0.0-beta5

Post upgrade of Devlake using Helm Chart, I encountered the following error.

Events:
  Type     Reason              Age                    From                     Message
  ----     ------              ----                   ----                     -------
  Normal   Scheduled           6m47s                  default-scheduler        Successfully assigned devlake/devlake-grafana-6554869b4-czhzw to ip-10-85-44-217.ap-south-1.compute.internal
  Warning  FailedAttachVolume  6m47s                  attachdetach-controller  Multi-Attach error for volume "pvc-fff59316-7f48-406d-abb1-9c02c0895571" Volume is already used by pod(s) devlake-grafana-79775f95f8-4ksq9
  Warning  FailedMount         2m28s (x2 over 4m44s)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[storage], unattached volumes=[storage kube-api-access-sh8m6 config]: timed out waiting for the condition
  Warning  FailedMount         10s                    kubelet                  Unable to attach or mount volumes: unmounted volumes=[storage], unattached volumes=[kube-api-access-sh8m6 config storage]: timed out waiting for the condition

Workaround:

➜  ~ k -n devlake scale deployment/devlake-grafana --replicas=0
deployment.apps/devlake-grafana scaled
➜  ~ k -n devlake scale deployment/devlake-grafana --replicas=1
deployment.apps/devlake-grafana scaled

[klesh]

@ZhangNing10 Please take a look at this.

[ZhangNing10]

hi @wnds , actually you can adjust the grafana pod update policy as per your infra setup. I guess you use something like the aws ebs as pvc storage, so only one node can mount the pvc at the same time. I have created pr https://github.com/apache/incubator-devlake-helm-chart/pull/277 which you can refer to