cloud-provider-openstack icon indicating copy to clipboard operation
cloud-provider-openstack copied to clipboard

Published 20 hours ago • verified publisher icon kubernetes

[cinder-csi-plugin] Snapshots not topology aware

Open huxcrux opened this issue 3 years ago • 3 comments

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened: When running a Kubernetes clusters spanning over multiple availability zones and storage is per zone snapshots restore fails if volume isn't created in the same zone as the snapshot exists in. I also noticed that the snapshot and snapshot content doesn't contain any information related to topology

Persistent volumes does always works as intended since we can use the allowedTopologies option:

Example of a volumesnapshot:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  annotations:
  creationTimestamp: "2022-07-19T09:15:56Z"
  finalizers:
  - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
  - snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  generation: 1
  name: snapshot-1
  namespace: default
  resourceVersion: "6853"
  uid: 7f70fe4b-8e05-4a16-94df-42a8423302dd
spec:
  source:
    persistentVolumeClaimName: www-web-0
  volumeSnapshotClassName: cinder-csi-snapshot
status:
  boundVolumeSnapshotContentName: snapcontent-7f70fe4b-8e05-4a16-94df-42a8423302dd
  creationTime: "2022-07-19T09:21:31Z"
  readyToUse: true

Example of a volumesnapshotcontent:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
  creationTimestamp: "2022-07-19T09:15:56Z"
  finalizers:
  - snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection
  generation: 1
  name: snapcontent-7f70fe4b-8e05-4a16-94df-42a8423302dd
  resourceVersion: "6851"
  uid: c683c20b-a786-4547-a601-22b6163cdbc7
spec:
  deletionPolicy: Delete
  driver: cinder.csi.openstack.org
  source:
    volumeHandle: 17a1b139-0cc8-43ea-b918-4c71eade931d
  volumeSnapshotClassName: cinder-csi-snapshot
  volumeSnapshotRef:
    apiVersion: snapshot.storage.k8s.io/v1
    kind: VolumeSnapshot
    name: snapshot-1
    namespace: default
    resourceVersion: "4042"
    uid: 7f70fe4b-8e05-4a16-94df-42a8423302dd
status:
  creationTime: 1658222491787310000
  readyToUse: true
  restoreSize: 1073741824
  snapshotHandle: e5c82fb5-c6a2-4705-9422-f3b19e643915

As long as the volume still exists you can get the zone from the volume, however if the volume used to create a snapshot from doesn't exists there is no way inside Kubernetes to tell what zone a snapshot exists in.

What you expected to happen: I expect the volume to be created in the same zone as the snapshot exists in every time. In order to do this I assume the snapshot somehow needs a topology reference?

How to reproduce it:

In the example below we will have a Kubernetes cluster spanning over 3 different availability zones each with separate storage and no cross mounting allowed.

  1. Have a kubernetes cluster spanning over multiple zones with separate storage (no cross mount allowed)
  2. Install cinder-csi-plugin
  3. Create storageclass that is topology aware
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: 4k
allowVolumeExpansion: true
allowedTopologies:
- matchLabelExpressions:
  - key: topology.cinder.csi.openstack.org/zone
    values:
    - sto1
    - sto2
    - sto3
parameters:
  type: 4k-IOPS
provisioner: cinder.csi.openstack.org
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
  1. Create a snapshotclass
apiVersion: snapshot.storage.k8s.io/v1
deletionPolicy: Delete
driver: cinder.csi.openstack.org
kind: VolumeSnapshotClass
metadata:
  name: cinder-csi-snapshot
parameters:
  force-create: "True"
  1. Create a volume (I created a statefulset to simplify things)
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: k8s.gcr.io/nginx-slim:0.8
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
  1. Create a snapshot from volume
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: snapshot-1
spec:
  volumeSnapshotClassName: cinder-csi-snapshot
  source:
    persistentVolumeClaimName: www-web-0
  1. Restore volume to a newly created volume (this step fails when the volume is created in another az than the snapshot exists in)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: snapshot-1-restore
spec:
  storageClassName: 4k
  dataSource:
    name: snapshot-1
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

Anything else we need to know?:

Environment:

  • openstack-cloud-controller-manager(or other related binary) version: docker.io/k8scloudprovider/cinder-csi-plugin:v1.22.0

    k8s.gcr.io/sig-storage/csi-attacher:v3.3.0 k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.1 k8s.gcr.io/sig-storage/csi-resizer:v1.3.0 k8s.gcr.io/sig-storage/livenessprobe:v2.5.0

    k8s.gcr.io/sig-storage/snapshot-controller:v4.2.1

  • OpenStack version: Train

  • Others: Kubernetes 1.22(.8) - doesn't seems to be relevant however is the version I used for all examples

huxcrux avatar Jul 20 '22 09:07 huxcrux

Thanks for detailed report, some comments/insights @bl0m1

I didn't try myself, but are we able to create such volume if the volume snapshot in other AZ through openstack API ? I found https://docs.openstack.org/api-ref/block-storage/v3/index.html?expanded=create-a-snapshot-detail,create-a-volume-detail#create-a-volume says;

To create a volume from an existing snapshot, specify the UUID of the volume snapshot. The volume is created in same availability zone and with same size as the snapshot.

so I doubt whether it's doable through openstack side? basically, the snapshot content need to be copied across AZ which not sure doable now .. if the openstack is eligible to do so, I think we can propose changes in CPO and even report to https://github.com/kubernetes-csi/external-snapshotter/

jichenjc avatar Jul 22 '22 07:07 jichenjc

Thanks for detailed report, some comments/insights @bl0m1

I didn't try myself, but are we able to create such volume if the volume snapshot in other AZ through openstack API ? I found https://docs.openstack.org/api-ref/block-storage/v3/index.html?expanded=create-a-snapshot-detail,create-a-volume-detail#create-a-volume says;

To create a volume from an existing snapshot, specify the UUID of the volume snapshot. The volume is created in same availability zone and with same size as the snapshot.

so I doubt whether it's doable through openstack side? basically, the snapshot content need to be copied across AZ which not sure doable now .. if the openstack is eligible to do so, I think we can propose changes in CPO and even report to https://github.com/kubernetes-csi/external-snapshotter/

Hello, I have tested to create a lot of volumes from a snapshot inside openstack it it works as intended (as described in the link you sent) and the new volume based on a snapshot seems to always be created in the correct zone.

Once a restore fails the following event is created:

28s         Warning   FailedAttachVolume       pod/snapshot-restore-test-7                      AttachVolume.Attach failed for volume "pvc-9a7be703-950c-4909-8481-33261ef9c364" : rpc error: code = Internal desc = [ControllerPublishVolume] Attach Volume failed with error failed to attach 94519796-68bf-4e88-a04d-18025d9a0ff1 volume to 44d341f0-ed3b-4b63-bbcd-bc4e5d316a44 compute: Bad request with: [POST https://ops.elastx.cloud:8774/v2.1/servers/44d341f0-ed3b-4b63-bbcd-bc4e5d316a44/os-volume_attachments], error message: {"badRequest": {"code": 400, "message": "Invalid input received: Invalid volume: Volume 94519796-68bf-4e88-a04d-18025d9a0ff1 status must be available or downloading to reserve, but the current status is creating. (HTTP 400) (Request-ID: req-b1395797-ca18-4057-b0a5-50cae1ddecce)"}}

When checking inside Openstack there is no reference at all to this volume, like it was never created or deleted once the snapshot restore process failed.

I created 10 volumes that restores the same snapshot and the a pod that attaches the volumes.

❯ kubectl get pvc
NAME                       STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
snapshot-restore-test-1    Pending                                                                        4k             3m15s
snapshot-restore-test-10   Pending                                                                        4k             3m14s
snapshot-restore-test-2    Pending                                                                        4k             3m15s
snapshot-restore-test-3    Pending                                                                        4k             3m15s
snapshot-restore-test-4    Pending                                                                        4k             3m15s
snapshot-restore-test-5    Bound     pvc-2d8d173a-cb94-41e3-bdad-e533915d1feb   1Gi        RWO            4k             3m15s
snapshot-restore-test-6    Pending                                                                        4k             3m14s
snapshot-restore-test-7    Bound     pvc-0f15ebb8-8c3a-4c48-bf00-b36aba7b8db7   1Gi        RWO            4k             3m14s
snapshot-restore-test-8    Pending                                                                        4k             3m14s
snapshot-restore-test-9    Bound     pvc-faccf6a9-320d-4b06-b85d-2822e01df2ef   1Gi        RWO            4k             3m14s

I collected logs (I was only able to find relevant logs inside the csi-provisioner and cinder-csi-plugin containers, In order to not make a to long response I created a gist with the relevant logs that can be found here: https://gist.github.com/bl0m1/78f2657ab327af10516fa4db75b847fd

huxcrux avatar Jul 26 '22 13:07 huxcrux

Hello, I have tested to create a lot of volumes from a snapshot inside openstack it it works as intended (as described in the link you sent) and the new volume based on a snapshot seems to always be created in the correct zone.

ok ,the original bug reported mentioned this step fails when the volume is created in another az than the snapshot exists in) because if the volume to be restored from snapshot and the snapshot itself are in different AZ , so you are saying above that you can restore a volume with snapshot from different AZ through openstack CLI/API (which is , without CPO) correct?

if so , there must be something wrong in the param we send or the CSI mechanism need optimization, so please confirm that with CLI (without CPO) only you can create volume whose source is snapshot in different AZ.. thanks

jichenjc avatar Jul 28 '22 03:07 jichenjc

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Oct 26 '22 03:10 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot avatar Nov 25 '22 04:11 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-triage-robot avatar Dec 25 '22 04:12 k8s-triage-robot

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot avatar Dec 25 '22 04:12 k8s-ci-robot