Kubernetes: `airbyte-volume-configs` PVC declared as ReadWriteOnce, but is used in multiple pods

[ball-hayden]

Environment

• Airbyte version: 0.40.6
• OS Version / Instance: Official Docker
• Deployment: Kubernetes
• Step where error happened: Deploy

Current Behavior

airbyte-cron deployment pods fail to start, because airbyte-volume-configs is already mounted by airbyte-server pod

Expected Behavior

A ReadWriteOnce PVC should not be used by multiple pods.

Logs

Multi-Attach error for volume "pvc-24380ad7-a93d-4a45-ae66-4dfee25947eb" Volume is already used by pod(s) airbyte-server-86d4c84855-2ng89

Steps to Reproduce

1. Use kustomize to deploy https://github.com/airbytehq/airbyte/tree/master/kube/overlays/stable-with-resource-limits
2. Observe error

Are you willing to submit a PR?

I'd need guidance as to how the team would like to handle this.

One option would be to use a ReadWriteMany, although this doesn't seem to be supported by many providers. Another option might be to have airbyte-cron contact airbyte-server to fetch config?

[dis-sid]

This one had me deleting the airbyte server pod multiple times until it gets scheduled on the same node as the airbyte-cron pod (ReadWriteOnce is accessible by multiple pods if they are on the same node). It makes upgrading the airbyte instance complicated.

I can't recommend using ReadWriteMany beacause of so few providers supporting it and neither using node affinity constraints because it makes the cluster more rigid. If both airbyte-cron and server need write access to it we can:

• Move the config into object storage
• Have both server and cron run into the same pod
• Like @ball-hayden proposed, have a service handling read/write access to the config volume

[StevenReitsma]

Weirdly enough the Helm chart doesn't seem to have this airbyte-volume-configs volume at all... https://github.com/airbytehq/airbyte/blob/master/charts/airbyte-cron/templates/deployment.yaml

What's this PVC used for anyway?

[Zeppier]

Weirdly enough the Helm chart doesn't seem to have this airbyte-volume-configs volume at all... https://github.com/airbytehq/airbyte/blob/master/charts/airbyte-cron/templates/deployment.yaml

But I see it here: https://github.com/airbytehq/airbyte/blob/master/kube/resources/volume-configs.yaml.

And we have the same problems now. If a restart happens we need to do a lot of workarounds to make it work. Any insights on why we need it at all?

[mozarik]

Yep have that problem too

  Normal   Scheduled           45m                 default-scheduler        Successfully assigned airbyte/airbyte-cron-765ccb6b44-9lb2j to gke-ase1-glo-data-1-e2-std-14-36-de-s-3e48975c-4xw8
  Warning  FailedAttachVolume  45m                 attachdetach-controller  Multi-Attach error for volume "pvc-cf321101-735f-47e1-ae5a-607a8c26fa1a" Volume is already used by pod(s) airbyte-server-8c464cd5f-tbc96
  Warning  FailedMount         16m (x3 over 40m)   kubelet                  Unable to attach or mount volumes: unmounted volumes=[airbyte-volume-configs], unattached volumes=[gcs-log-creds-volume kube-api-access-lkqfk airbyte-volume-configs]: timed out waiting for the condition
  Warning  FailedMount         13m (x3 over 34m)   kubelet                  Unable to attach or mount volumes: unmounted volumes=[airbyte-volume-configs], unattached volumes=[kube-api-access-lkqfk airbyte-volume-configs gcs-log-creds-volume]: timed out waiting for the condition
  Warning  FailedMount         20s (x14 over 43m)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[airbyte-volume-configs], unattached volumes=[airbyte-volume-configs gcs-log-creds-volume kube-api-access-lkqfk]: timed out waiting for the condition

Currently i delete that volumt mount

[mozarik]

Is it related CSI Driver (GKE) here https://airbyte.gitbook.io/airbyte/deploying-airbyte/on-kubernetes#persistent-storage-on-gke-regional-cluster

[obraginskaya]

Hey, we got the same issue with kustomize deployment. Any ideas? Maybe Deployment should be changed to StatefulSet as it was proposed here https://github.com/airbytehq/airbyte/issues/9133#issuecomment-1127667156?

[Kopiczek]

Is there any update to this?

[lydialimsetel]

Facing same issue here. So, I ended up commenting out the whole volumeMounts and volume section in cron.yaml. So now airbyte-cron doesn't go and fight with airbyte-server for the volume. The volume is likely not useful anyway.

  #     volumeMounts:
  #       - name: airbyte-volume-configs
  #         mountPath: /configs
  #       - name: gcs-log-creds-volume
  #         mountPath: /secrets/gcs-log-creds
  #         readOnly: true
  # volumes:
  #   - name: airbyte-volume-configs
  #     persistentVolumeClaim:
  #       claimName: airbyte-volume-configs
  #   - name: gcs-log-creds-volume
  #     secret:
  #       secretName: gcs-log-creds