azuredisk-csi-driver icon indicating copy to clipboard operation
azuredisk-csi-driver copied to clipboard

Published 20 hours ago • verified publisher icon kubernetes-sigs

Azure Disk in a Multi Region Kubernetes cluster is provisioned in the wrong Region

Open MrAbaddon opened this issue 1 year ago • 4 comments

What happened:

I have a self managed cluster in Azure in 2 different regions. North Europe and west Europe. 1 control plane and 3 worker nodes in North Europe. 1 control plane and 3 worker nodes in West Europe. I apply a PVC manifest and a Statefullset with node selector for westeurope and I see that the azure disk is provisioned in northeurope.

cat <<EOF | kubectl apply --filename=-
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: statefulset-azuredisk
  namespace: default
  labels:
    app: nginx
spec:
  podManagementPolicy: Parallel  # default is OrderedReady
  serviceName: statefulset-azuredisk
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        kubernetes.io/dedicatedhost: host-02 ### host-02 is an azure dedicated host with 3VM for worker nodes in northeurope
      containers:
        - name: statefulset-azuredisk
          image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
          command:
            - "/bin/bash"
            - "-c"
            - set -euo pipefail; while true; do echo $(date) >> /mnt/azuredisk/outfile; sleep 1; done
          volumeMounts:
            - name: persistent-storage
              mountPath: /mnt/azuredisk
              readOnly: false
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: nginx
  volumeClaimTemplates:
    - metadata:
        name: persistent-storage
      spec:
        storageClassName: managed-csi
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi
EOF

Looking at the logs in the controller:

I0528 21:30:38.162910       1 utils.go:105] GRPC call: /csi.v1.Controller/CreateVolume
I0528 21:30:38.162962       1 utils.go:106] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westeurope-1","topology.kubernetes.io/zone":"westeurope-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westeurope-1","topology.kubernetes.io/zone":"westeurope-1"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-669b1eae-d04b-485a-9987-8f95d86b2a10","parameters":{"csi.storage.k8s.io/pv/name":"pvc-669b1eae-d04b-485a-9987-8f95d86b2a10","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0528 21:30:38.163158       1 controllerserver.go:219] begin to create azure disk(pvc-669b1eae-d04b-485a-9987-8f95d86b2a10) account type(StandardSSD_LRS) rg(xxxxxxxxx) location(northeurope) size(10) diskZone() maxShares(0)
I0528 21:30:38.163177       1 azure_managedDiskController.go:111] azureDisk - creating new managed Name:pvc-669b1eae-d04b-485a-9987-8f95d86b2a10 StorageAccountType:StandardSSD_LRS Size:10
I0528 21:30:40.948927       1 azure_managedDiskController.go:304] azureDisk - created new MD Name:pvc-669b1eae-d04b-485a-9987-8f95d86b2a10 StorageAccountType:StandardSSD_LRS Size:10
I0528 21:30:40.948954       1 controllerserver.go:317] create azure disk(pvc-669b1eae-d04b-485a-9987-8f95d86b2a10) account type(StandardSSD_LRS) rg(xxxxxxxxx) location(northeurope) size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-669b1eae-d04b-485a-9987-8f95d86b2a10 kubernetes.io-created-for-pvc-name:persistent-storage-statefulset-azuredisk-0 kubernetes.io-created-for-pvc-namespace:default]) successfully
I0528 21:30:40.949006       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=2.785803836 request="azuredisk_csi_driver_controller_create_volume" resource_group="xxxxxxxxx" subscription_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" source="disk.csi.azure.com" volumeid="/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-669b1eae-d04b-485a-9987-8f95d86b2a10" result_code="succeeded"
I0528 21:30:40.949017       1 utils.go:112] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":""}}],"capacity_bytes":10737418240,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-669b1eae-d04b-485a-9987-8f95d86b2a10","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-669b1eae-d04b-485a-9987-8f95d86b2a10"}}

We can see that in the Request it detects the topology "topology.disk.csi.azure.com/zone":"westeurope-1","topology.kubernetes.io/zone":"westeurope-1". Thats correct. But the controller then creates in location(northeurope) size(10) diskZone(). northeurope without an AZ and this is wrong.

If I change the node selector to northeurope, the azure disk is provisioned in northeurope and everything works good.

cat <<EOF | kubectl apply --filename=-
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: statefulset-azuredisk
  namespace: default
  labels:
    app: nginx
spec:
  podManagementPolicy: Parallel  # default is OrderedReady
  serviceName: statefulset-azuredisk
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        kubernetes.io/dedicatedhost: host-01 ### host-01 is an azure dedicated host with 3VM for worker nodes in westeurope
      containers:
        - name: statefulset-azuredisk
          image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
          command:
            - "/bin/bash"
            - "-c"
            - set -euo pipefail; while true; do echo $(date) >> /mnt/azuredisk/outfile; sleep 1; done
          volumeMounts:
            - name: persistent-storage
              mountPath: /mnt/azuredisk
              readOnly: false
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: nginx
  volumeClaimTemplates:
    - metadata:
        name: persistent-storage
      spec:
        storageClassName: managed-csi
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi
EOF

And the Logs:

I0528 21:38:38.132216       1 utils.go:105] GRPC call: /csi.v1.Controller/CreateVolume
I0528 21:38:38.132724       1 utils.go:106] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"northeurope-1","topology.kubernetes.io/zone":"northeurope-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"northeurope-1","topology.kubernetes.io/zone":"northeurope-1"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-6083de42-5d50-4b6b-84ca-2812374d9a60","parameters":{"csi.storage.k8s.io/pv/name":"pvc-6083de42-5d50-4b6b-84ca-2812374d9a60","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"blockchain","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}
I0528 21:38:38.132827       1 controllerserver.go:219] begin to create azure disk(pvc-6083de42-5d50-4b6b-84ca-2812374d9a60) account type(StandardSSD_LRS) rg(xxxxxxxxx) location(northeurope) size(10) diskZone(northeurope-1) maxShares(0)
I0528 21:38:38.132839       1 azure_managedDiskController.go:111] azureDisk - creating new managed Name:pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 StorageAccountType:StandardSSD_LRS Size:10
I0528 21:38:41.020827       1 azure_managedDiskController.go:304] azureDisk - created new MD Name:pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 StorageAccountType:StandardSSD_LRS Size:10
I0528 21:38:41.020855       1 controllerserver.go:317] create azure disk(pvc-6083de42-5d50-4b6b-84ca-2812374d9a60) account type(StandardSSD_LRS) rg(xxxxxxxxx) location(northeurope) size(10) tags(map[kubernetes.io-created-for-pv-name:pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 kubernetes.io-created-for-pvc-name:persistent-storage-statefulset-azuredisk-0 kubernetes.io-created-for-pvc-namespace:blockchain]) successfully
I0528 21:38:41.020879       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=2.888026849 request="azuredisk_csi_driver_controller_create_volume" resource_group="xxxxxxxxx" subscription_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" source="disk.csi.azure.com" volumeid="/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60" result_code="succeeded"
I0528 21:38:41.020890       1 utils.go:112] GRPC response: {"volume":{"accessible_topology":[{"segments":{"topology.disk.csi.azure.com/zone":"northeurope-1"}}],"capacity_bytes":10737418240,"content_source":{"Type":null},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6083de42-5d50-4b6b-84ca-2812374d9a60","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"blockchain","requestedsizegib":"10","skuName":"StandardSSD_LRS"},"volume_id":"/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60"}}
I0528 21:38:42.288867       1 utils.go:105] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0528 21:38:42.289473       1 utils.go:106] GRPC request: {"node_id":"xxxxxxxxx-host-01-worker-02","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-6083de42-5d50-4b6b-84ca-2812374d9a60","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"blockchain","requestedsizegib":"10","skuName":"StandardSSD_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1716913765467-4400-disk.csi.azure.com"},"volume_id":"/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60"}
I0528 21:38:42.795078       1 controllerserver.go:429] GetDiskLun returned: cannot find Lun for disk pvc-6083de42-5d50-4b6b-84ca-2812374d9a60. Initiating attaching volume /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 to node xxxxxxxxx-host-01-worker-02 (vmState Succeeded).
I0528 21:38:42.795103       1 controllerserver.go:456] Trying to attach volume /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 to node xxxxxxxxx-host-01-worker-02
I0528 21:38:42.795115       1 azure_controller_common.go:212] wait 1000ms for more requests on node xxxxxxxxx-host-01-worker-02, current disk attach: /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60
I0528 21:38:43.795315       1 azure_controller_common.go:226] Trying to attach volume /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 lun 0 to node xxxxxxxxx-host-01-worker-02, diskMap len:1, map[/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/xxxxxxxxx/providers/microsoft.compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60:0xc000207e80]
I0528 21:38:43.795346       1 azure_controller_standard.go:103] azureDisk - update(xxxxxxxxx): vm(xxxxxxxxx-host-01-worker-02) - attach disk list(map[/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/xxxxxxxxx/providers/microsoft.compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60:0xc000207e80])
I0528 21:38:44.440915       1 azure_controller_standard.go:116] azureDisk - update(xxxxxxxxx): vm(xxxxxxxxx-host-01-worker-02) - attach disk list(map[/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/xxxxxxxxx/providers/microsoft.compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60:0xc000207e80]) returned with <nil>
I0528 21:38:48.663923       1 azure_controller_standard.go:126] DeleteCacheForNode(xxxxxxxxx-host-01-worker-02) successfully
I0528 21:38:48.663941       1 azure_controller_standard.go:269] updateCache(xxxxxxxxx-host-01-worker-02) successfully
I0528 21:38:48.664012       1 azure_controller_common.go:504] azureDisk - find disk: lun 0 name pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 uri /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60
I0528 21:38:48.664027       1 controllerserver.go:465] Attach operation successful: volume /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 attached to node xxxxxxxxx-host-01-worker-02.
I0528 21:38:48.664037       1 controllerserver.go:489] attach volume /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60 to node xxxxxxxxx-host-01-worker-02 successfully
I0528 21:38:48.664056       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=6.157552193 request="azuredisk_csi_driver_controller_publish_volume" resource_group="xxxxxxxxx" subscription_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" source="disk.csi.azure.com" volumeid="/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/xxxxxxxxx/providers/Microsoft.Compute/disks/pvc-6083de42-5d50-4b6b-84ca-2812374d9a60" node="xxxxxxxxx-host-01-worker-02" result_code="succeeded"
I0528 21:38:48.664064       1 utils.go:112] GRPC response: {"publish_context":{"LUN":"0"}}

We can see that in request it detects topology "topology.disk.csi.azure.com/zone":"northeurope-1","topology.kubernetes.io/zone":"northeurope-1". Thats correct. And the controller creates in correct location: begin to create azure disk(pvc-6083de42-5d50-4b6b-84ca-2812374d9a60) account type(StandardSSD_LRS) rg(xxxxxxxxx) location(northeurope) size(10) diskZone(northeurope-1) maxShares(0).

What you expected to happen:

The correct azure disk is provisioned in the correct region and zone

Environment:

  • CSI Driver version: image: mcr.microsoft.com/oss/kubernetes-csi/azuredisk-csi:v1.30.0
  • Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"29", GitVersion:"v1.29.5", GitCommit:"59755ff595fa4526236b0cc03aa2242d941a5171", GitTreeState:"clean", BuildDate:"2024-05-14T10:39:39Z", GoVersion:"go1.21.9", Compiler:"gc", Platform:"linux/amd64"} WARNING: version difference between client (1.23) and server (1.29) exceeds the supported minor version skew of +/-1

MrAbaddon avatar May 28 '24 22:05 MrAbaddon

if you set volumeBindingMode: WaitForFirstConsumer in storage class, the pod would be scheduled to the node first, and then pvc creation would happen, that could make sure you have the right region together with pod. Otherwise how does the disk controller know which region you want to create a disk pv?

andyzhangx avatar Jun 07 '24 09:06 andyzhangx

This is my StorageClasse manifest.

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: managed-csi annotations: storageclass.kubernetes.io/is-default-class: "true" provisioner: disk.csi.azure.com parameters: skuName: StandardSSD_LRS # available values: StandardSSD_LRS, StandardSSD_ZRS, Premium_LRS, Premium_ZRS, etc. reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true

So I've seted volumeBindingMode: WaitForFirstConsumer. I can see in logs that the request is made for the correct region:

I0528 21:30:38.162962 1 utils.go:106] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.disk.csi.azure.com/zone":"westeurope-1","topology.kubernetes.io/zone":"westeurope-1"}}],"requisite":[{"segments":{"topology.disk.csi.azure.com/zone":"westeurope-1","topology.kubernetes.io/zone":"westeurope-1"}}]},"capacity_range":{"required_bytes":10737418240},"name":"pvc-669b1eae-d04b-485a-9987-8f95d86b2a10","parameters":{"csi.storage.k8s.io/pv/name":"pvc-669b1eae-d04b-485a-9987-8f95d86b2a10","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-azuredisk-0","csi.storage.k8s.io/pvc/namespace":"default","skuName":"StandardSSD_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":7}}]}

The pod is created in a node in westeurope but the disk is requested where the controller is based, in this case northeurope:

I0528 21:38:38.132827 1 controllerserver.go:219] begin to create azure disk(pvc-6083de42-5d50-4b6b-84ca-2812374d9a60) account type(StandardSSD_LRS) rg(xxxxxxxxx) location(northeurope) size(10) diskZone(northeurope-1) maxShares(0)

Im using the helm charts to install the driver. So as a work around I'm using the values to force the controller to be replica 1 and with node selectors I'm placing the controller in a node that is located in a region where I want to place the disk. A bit manual but works...

MrAbaddon avatar Jun 11 '24 17:06 MrAbaddon

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Sep 09 '24 17:09 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot avatar Oct 09 '24 18:10 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-triage-robot avatar Nov 08 '24 18:11 k8s-triage-robot

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot avatar Nov 08 '24 18:11 k8s-ci-robot