TiKV region balancing does not work

[conanoc]

Bug Report

What did you do?

1. Create a TiKV cluster on k8s. tikv storage size is 2Ti with 30 replicas.
2. Insert 10TB of data.
3. Wait until leader/region balancing finishes.
4. Insert another 10TB of data

What did you expect to see?

Leader/region balancing works.

What did you see instead?

Leader balancing worked, but region balancing did not work.

PD panel shows that there are 10 LowSpace stores.

Many stores have enough available storage.

What version of PD are you using (pd-server -V)?

v8.5.2

[bufferflies]

Can you give me the clinic or the pd->balance metrics well?

[conanoc]

Here's the pd->balance panel. I don't know what "clinic" is.

[bufferflies]

Here's the pd->balance panel. I don't know what "clinic" is.

you can use this tools to upload the metrics to our metrics platforms https://docs.pingcap.com/tidb/stable/clinic-introduction/

[bufferflies]

What's your store label, location setting, and placement rules?

[conanoc]

I used location-labels = ["kubernetes.io/hostname"]. I have 15 worker nodes in the k8s cluster. This is the cluster yaml file I used.

---
apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: basic
spec:
  version: v8.5.2
  timezone: "+9:00"
  pvReclaimPolicy: Delete
  enableDynamicConfiguration: true
  configUpdateStrategy: RollingUpdate
  discovery: {}
  helper:
    image: alpine:3.16.0
  pd:
    baseImage: pingcap/pd
    maxFailoverCount: 3
    replicas: 3
    requests:
      storage: 10Gi
    config: |
      [replication]
        max-replicas = 3
        location-labels = ["kubernetes.io/hostname"]
  tikv:
    baseImage: pingcap/tikv
    maxFailoverCount: 10
    recoverFailover: true
    evictLeaderTimeout: 3m
    replicas: 30
    requests:
      storage: 2Ti
    config: |
      memory-usage-limit = "4GB"
      [log.file]
        filename = "/var/lib/tikv/tikv.log"
        max-days = 3
      [storage]
        reserve-space = "0MB"
        api-version = 2
        enable-ttl = true
        [storage.block-cache]
          capacity = "1GB"
  tidb:
    baseImage: pingcap/tidb
    maxFailoverCount: 2
    replicas: 2
    config: {}

[bufferflies]

Why does the used_size + available_size != capacity in your cluster, such as store id 1014? One possible reason is that the other tikv shares one disk, or the disk has extra data that does not belong to the tikv.

[conanoc]

You are right. There are 15 k8s worker nodes and 30 tikv stores. So, there are two tikv stores in one worker node on average.

[bufferflies]

You are right. There are 15 k8s worker nodes and 30 tikv stores. So, there are two tikv stores in one worker node on average.

In this case, you need to set the tikv's capacity manually.

[conanoc]

Isn't it better for the TiKV node to calculate its storage capacity based on the size of the PV instead of using the machine's disk size or adjusting it manually?

[bufferflies]

Isn't it better for the TiKV node to calculate its storage capacity based on the size of the PV instead of using the machine's disk size or adjusting it manually?

TIKV will collect the PV details if your PV is normal. You can log in to the TikV pods and see the disk capacity. If the capaciity is not equal the PV instance, it indicates that your pv environment is wrong.

[conanoc]

Below is what I get from pvc. Please tell me more about how to check which is wrong.

% kubectl -n tidb-cluster describe pvc tikv-basic-tikv-0
Name:          tikv-basic-tikv-0
Namespace:     tidb-cluster
StorageClass:  local-path
Status:        Bound
Volume:        pvc-054d028d-8c59-4c2d-a8df-a02bf8897673
Labels:        app.kubernetes.io/component=tikv
               app.kubernetes.io/instance=basic
               app.kubernetes.io/managed-by=tidb-operator
               app.kubernetes.io/name=tidb-cluster
               tidb.pingcap.com/cluster-id=7561722164261267549
               tidb.pingcap.com/pod-name=basic-tikv-0
               tidb.pingcap.com/store-id=1014
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               tidb.pingcap.com/pod-name: basic-tikv-0
               volume.beta.kubernetes.io/storage-provisioner: rancher.io/local-path
               volume.kubernetes.io/selected-node: ctikvnode007.kv
               volume.kubernetes.io/storage-provisioner: rancher.io/local-path
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      2Ti
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       basic-tikv-0
Events:        <none>

[bufferflies]

In your pvc crd, the storage capacity is 2TI. But on the PD side, the TikV storage capacity is 20T. You can select one tikv pods to login, and see the actual size.

[conanoc]

You can select one tikv pods to login, and see the actual size.

I don't know how I can see the actual size of that tikv store. You mentioned that if the actual size is not equal to the capacity of the pv, then we can say "the pv environment is wrong". I would still wonder what is wrong. Is it a bug in rancher.io/local-path? Or is it a bug in tikv, pd, or tidb operator? Or is the cluster yaml file I wrote the problem?

[bufferflies]

In my env, I can follow this steps to show the capacity of tikv data directory :

[conanoc]

I see. In the case of local-path storage, several pods will share the disk space. So, the available disk space will not be the appropriate measure for the storage size of a tikv node. After I set tikv.limits.storage to 2Ti, it seems the cluster uses that value as the storage size of tikv nodes. What about setting the default value of tikv.limits.storage to tikv.requests.storage?

The region rebalancing progressed as expected after I changed the tikv.limits.storage value. In the following dashboard, I changed the value at around 15:20.

[bufferflies]

generally, we don't suggest the user to share one disk in prod env

[conanoc]

Is there any reason? Considering using local storage on k8s, sharing one disk would be inevitable.