Need detailed steps on this recovery

[coderplay]

File: /release-1.0/en/troubleshoot.md

If a StorageClass exists in the cluster but the available PV is insufficient, you need to add PV resources correspondingly.

How? The PV is supposed to be managed by tidb-operator, how can we manually add it and let tidb-operator honor it?

[DanielZhangQD]

@coderplay The PV is not managed by tidb-operator, it's managed by the volume provisioner, e.g. local-volume-provisioner for local storage, or EBS CSI for EBS. Even the PVC is not managed by tidb-operator, it's created automatically by the Kubernetes native statefulset controller.

[coderplay]

@DanielZhangQD The way I understand it as managed is that the PV is provisioned and maintained by the tidb-operator. TiDB-operator adds many labels and annotations to the PV it manages. See below for example:

kind: PersistentVolume
metadata:
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/migrated-to: disk.csi.azure.com
    pv.kubernetes.io/provisioned-by: kubernetes.io/azure-disk
    tidb.pingcap.com/pod-name: tidb-tikv-5
  creationTimestamp: "2022-03-12T08:22:38Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/disk-csi-azure-com
  labels:
    app.kubernetes.io/component: tikv
    app.kubernetes.io/instance: tidb
    app.kubernetes.io/managed-by: tidb-operator
    app.kubernetes.io/name: tidb-cluster
    app.kubernetes.io/namespace: tidb
    tidb.pingcap.com/cluster-id: "7062173269355868939"
    tidb.pingcap.com/store-id: "10692859"
  name: pvc-126420ef-5011-403c-bcb1-6535e4f5f34b
  resourceVersion: "52568853"
  uid: 2ee5ebd4-3eac-4329-b9ed-31b5ef796b2a
spec:
  accessModes:
  - ReadWriteOnce
  azureDisk:
    cachingMode: None
    diskName: pvc-126420ef-5011-403c-bcb1-6535e4f5f34b
    diskURI: /subscriptions/bde8047c-c146-4e29-ae5f-1a5766ba0d7e/resourceGroups/mc_dev-azure-eastus-tidb1/providers/Microsoft.Compute/disks/pvc-126420ef-5011-403c-bcb1-6535e4f5f34b
    fsType: ""
    kind: Managed
    readOnly: false
  capacity:
    storage: 1536Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: tikv-tidb-tikv-5
    namespace: tidb
    resourceVersion: "52567290"
    uid: 126420ef-5011-403c-bcb1-6535e4f5f34b
  mountOptions:
  - defaults
  - nodelalloc
  - noatime
  - discard
  - nobarrier
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.disk.csi.azure.com/zone
          operator: In
          values:
          - eastus-3
  persistentVolumeReclaimPolicy: Retain
  storageClassName: tidb-raft-storage-class
  volumeMode: Filesystem
status:
  phase: Bound

If I add a normal PV manually w/o any specific labels/annotations, how come the tidb-operator be able to recognize it?

[DanielZhangQD]

@coderplay It recognizes it through the Pod/PVC/PV bindings, the PVs are managed by the volume provisioners, TiDB Operator just updates the PV to add the labels after it's bound with the PVC that's for the Pod of the TiDB cluster.

[DanielZhangQD]

@coderplay Do you have any more questions about this issue?