SPC(disk replacement): steps to replace disk in SPC based pools

[mittachaitu]

How can we replace disk in cStor pool was created using SPC?

[mittachaitu]

Replace a Disk in cStor pool

0. Creating a cStor Pool

• Show the disks. kubectl get bd -n openebs --show-labels

• Create a claim.

apiVersion: openebs.io/v1alpha1
kind: StoragePoolClaim
metadata:
  name: cstor-disk-pool
spec:
  name: cstor-disk-pool
  type: disk
  poolSpec:
    poolType: mirrored
  blockDevices:
    blockDeviceList:

• Apply the claim. kubectl apply -f spc.yaml

• Verify the claim. kubectl get csp

• Verify pool creation on appropriate nodes. kubectl get pod -n openebs -o wide

• Create a StorageClass that uses the new claim.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-standard
  annotations:
    openebs.io/cas-type: cstor
    cas.openebs.io/config: |
      - name: StoragePoolClaim
        value: "cstor-disk-pool"

provisioner: openebs.io/provisioner-iscsi

1. Replacing a disk

• Exec into the pod associated with the disk that needs to be replace. kubectl exec -it cstor-disk-pool-nq1i-7c686f8cb9-zbc4n -n openebs -ccstor-pool-mgmt bash

• Use zpool command-line utility to check the disk status. zpool status

• Find the device node for the replacement disk. kubectl get bd -n openebs blockdevice-bd952ccaa0638d1a5f7ab334e65d7aad -o yaml

• e.g.

...
  devlinks:
  - kind: by-id
    links:
    - /dev/disk/by-id/scsi-0Google_PersistentDisk_cstor-it-disk-10
    - /dev/disk/by-id/google-cstor-it-disk-10
...

• Create a BlockDeviceClaim for the new disk.

apiVersion: openebs.io/v1alpha1
kind: BlockDeviceClaim
metadata:
  finalizers:
  - storagepoolclaim.openebs.io/finalizer
  Labels:
     # Put the name of your SPC
    openebs.io/storage-pool-claim: cstor-disk-pool
  # Put the name of bdc as bdc-<bd-uid>
  # This 'bd-uid' is the uid of new BD that we are going to use
  # The UID can be found via 'kubectl get bd <bd-name> -o yaml' 
  name: bdc-2a1e53a2-d06e-11e9-bfdd-42010a8000a5
  # If you have installed openebs in a different namespaces then put 
  # then put that namespace here.
  namespace: openebs
  ownerReferences:
  - apiVersion: openebs.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: StoragePoolClaim
    # Put the name of your SPC
    name: cstor-disk-pool
    # Put the UID of your SPC
    uid: 70df195f-d08d-11e9-bfdd-42010a8000a5
Spec:
  # Put the block device name that you are going to use as a replacement
  blockDeviceName: blockdevice-bd952ccaa0638d1a5f7ab334e65d7aad 
  # This is the host name of the new block device that we are going to
  # for replacement.
  hostName: gke-cstor-it-pool-1-bdf8ee93-tv69 
  resources:
    Requests:
       # Put the capacity of BD in GB here. Disks greater than 
       # or equal to this size will match.
      storage: 100G

• Apply the above bdc. kubectl apply -f bdc.yaml

• Verify whether status of claim(it should be in Bound) kubectl get bdc -n openebs

• Use zpool to replace the disk.

zpool replace cstor-7108b3c7-d08d-11e9-bfdd-42010a8000a5  /dev/disk/by-id/scsi-0Google_PersistentDisk_cstor-it-disk-9 /dev/disk/by-id/scsi-0Google_PersistentDisk_cstor-it-disk-10

where cstor-7108b3c7-d08d-11e9-bfdd-42010a8000a5 is the pool name, /dev/disk/by-id/scsi-0Google_PersistentDisk_cstor-it-disk-9 is the old disk, and /dev/disk/by-id/scsi-0Google_PersistentDisk_cstor-it-disk-10 is the replacement.

• Monitor progress. zpool status

Now we need to replace the blockdevice name in CSP and SPC resource to intact with replaced blockdevice

• Disable reconciliation on SPC By adding annotation reconcile.openebs.io/disable: true on SPC we can disable reconciliation of SPC kubectl edit spc <spc_name>

• Replace blockdevice on CSP Replace the blockdevice name and device link entry with new blockdevice name and device link using kubectl edit csp <csp_name> Note: csp_name will be the pool name where we performed replacement.

• Verify whether CSP has new blockdevice kubectl get csp <csp_name> -o yaml

• Remove device link entry and replace with new device. kubectl edit spc cstor-disk-pool

• Remove the block device claim for the replaced disk. kubectl delete bdc <old_blockdevice_claim_name> -n openebs after this remove finalizer entry on storagepoolclaim.openebs.io/finalizer the same bdc.

• Enable Reconciliation on SPC Enable the reconciliation on SPC by removing the annotation reconcile.openebs.io/disable: true kubectl edit spc <spc_name>