kubeblocks
kubeblocks copied to clipboard
Published
20 hours ago •
apecloud
apecloud
[BUG]VolumeExpansion ops of polardbx cn/cdc is always running
➜ ~ kbcli version Kubernetes: v1.27.3-gke.100 KubeBlocks: 0.7.0-alpha.18 kbcli: 0.7.0-alpha.18
The ops status is always running, the pv size has changed but pvc size does not change
➜ ~ helm repo add kubeblocks https://apecloud.github.io/helm-charts
^@^@"kubeblocks" has been added to your repositories
➜ ~ helm upgrade --install polardbx kubeblocks/polardbx --version 0.7.0-alpha.18
Release "polardbx" does not exist. Installing it now.
NAME: polardbx
LAST DEPLOYED: Tue Sep 26 16:46:20 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thanks for installing PolarDB-X using KubeBlocks!
1. Run the following command to create your first PolarDB-X cluster:
kbcli cluster create pxc --cluster-definition polardbx
2. Port-forward service to localhost and connect to PolarDB-X cluster:
kubectl port-forward svc/pxc-cn 3306:3306 mysql -h127.0.0.1 -upolardbx_root
➜ ~ kbcli cluster create test --cluster-definition=polardbx
Info: --cluster-version is not specified, ClusterVersion polardbx-v1.4.1 is applied by default
Cluster test created
➜ ~ kbcli cluster describe test
Name: test Created Time: Sep 26,2023 16:47 UTC+0800
NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY
default polardbx polardbx-v1.4.1 Updating Delete
Endpoints:
COMPONENT MODE INTERNAL EXTERNAL
gms ReadWrite test-gms.default.svc.cluster.local:3306 <none>
test-gms.default.svc.cluster.local:9104
dn ReadWrite test-dn.default.svc.cluster.local:3306 <none>
cn ReadWrite test-cn.default.svc.cluster.local:3306 <none>
test-cn.default.svc.cluster.local:9104
cdc ReadWrite test-cdc.default.svc.cluster.local:3306 <none>
test-cdc.default.svc.cluster.local:9104
Topology:
COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME
cdc test-cdc-0 <none> Running us-central1-c gke-yjtest-default-pool-16b6e83c-hsgn/10.128.0.37 Sep 26,2023 16:47 UTC+0800
cn test-cn-0 <none> Running us-central1-c gke-yjtest-default-pool-16b6e83c-hsgn/10.128.0.37 Sep 26,2023 16:47 UTC+0800
dn test-dn-0 leader Running us-central1-c gke-yjtest-default-pool-16b6e83c-xqs0/10.128.0.39 Sep 26,2023 16:47 UTC+0800
dn test-dn-1 follower Running us-central1-c gke-yjtest-default-pool-16b6e83c-pbkd/10.128.0.26 Sep 26,2023 16:47 UTC+0800
dn test-dn-2 follower Running us-central1-c gke-yjtest-default-pool-16b6e83c-rxv2/10.128.0.38 Sep 26,2023 16:54 UTC+0800
gms test-gms-0 leader Running us-central1-c gke-yjtest-default-pool-16b6e83c-xqs0/10.128.0.39 Sep 26,2023 16:47 UTC+0800
gms test-gms-1 follower Running us-central1-c gke-yjtest-default-pool-16b6e83c-pbkd/10.128.0.26 Sep 26,2023 16:47 UTC+0800
gms test-gms-2 follower Running us-central1-c gke-yjtest-default-pool-16b6e83c-rxv2/10.128.0.38 Sep 26,2023 16:47 UTC+0800
Resources Allocation:
COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS
gms false 1 / 1 1Gi / 1Gi data:21Gi kb-default-sc
dn false 1 / 1 1Gi / 1Gi data:21Gi kb-default-sc
cn false 1 / 1 1Gi / 1Gi data:21Gi kb-default-sc
cdc false 1 / 1 1Gi / 1Gi data:20Gi kb-default-sc
Images:
COMPONENT TYPE IMAGE
gms gms polardbx/polardbx-engine-2.0:latest
dn dn polardbx/polardbx-engine-2.0:latest
cn cn polardbx/polardbx-sql:latest
cdc cdc polardbx/polardbx-cdc:latest
Show cluster events: kbcli cluster list-events -n default test
➜ ~ kbcli cluster describe test
Name: test Created Time: Sep 26,2023 16:47 UTC+0800
NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY
default polardbx polardbx-v1.4.1 Updating Delete
Endpoints:
COMPONENT MODE INTERNAL EXTERNAL
gms ReadWrite test-gms.default.svc.cluster.local:3306 <none>
test-gms.default.svc.cluster.local:9104
dn ReadWrite test-dn.default.svc.cluster.local:3306 <none>
cn ReadWrite test-cn.default.svc.cluster.local:3306 <none>
test-cn.default.svc.cluster.local:9104
cdc ReadWrite test-cdc.default.svc.cluster.local:3306 <none>
test-cdc.default.svc.cluster.local:9104
Topology:
COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME
cdc test-cdc-0 <none> Running us-central1-c gke-yjtest-default-pool-16b6e83c-hsgn/10.128.0.37 Sep 26,2023 16:47 UTC+0800
cn test-cn-0 <none> Running us-central1-c gke-yjtest-default-pool-16b6e83c-hsgn/10.128.0.37 Sep 26,2023 16:47 UTC+0800
dn test-dn-0 leader Running us-central1-c gke-yjtest-default-pool-16b6e83c-xqs0/10.128.0.39 Sep 26,2023 16:47 UTC+0800
dn test-dn-1 follower Running us-central1-c gke-yjtest-default-pool-16b6e83c-pbkd/10.128.0.26 Sep 26,2023 16:47 UTC+0800
dn test-dn-2 follower Running us-central1-c gke-yjtest-default-pool-16b6e83c-rxv2/10.128.0.38 Sep 26,2023 16:54 UTC+0800
gms test-gms-0 leader Running us-central1-c gke-yjtest-default-pool-16b6e83c-xqs0/10.128.0.39 Sep 26,2023 16:47 UTC+0800
gms test-gms-1 follower Running us-central1-c gke-yjtest-default-pool-16b6e83c-pbkd/10.128.0.26 Sep 26,2023 16:47 UTC+0800
gms test-gms-2 follower Running us-central1-c gke-yjtest-default-pool-16b6e83c-rxv2/10.128.0.38 Sep 26,2023 16:47 UTC+0800
Resources Allocation:
COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS
gms false 1 / 1 1Gi / 1Gi data:21Gi kb-default-sc
dn false 1 / 1 1Gi / 1Gi data:21Gi kb-default-sc
cn false 1 / 1 1Gi / 1Gi data:21Gi kb-default-sc
cdc false 1 / 1 1Gi / 1Gi data:20Gi kb-default-sc
Images:
COMPONENT TYPE IMAGE
gms gms polardbx/polardbx-engine-2.0:latest
dn dn polardbx/polardbx-engine-2.0:latest
cn cn polardbx/polardbx-sql:latest
cdc cdc polardbx/polardbx-cdc:latest
Show cluster events: kbcli cluster list-events -n default test
➜ ~ kbcli cluster volume-expand test --components cn --volume-claim-templates data --storage 21Gi
Please type the name again(separate with white space when more than one): test
OpsRequest test-volumeexpansion-z5hcc created successfully, you can view the progress:
kbcli cluster describe-ops test-volumeexpansion-z5hcc -n default
➜ ~ kbcli cluster describe-ops test-volumeexpansion-z5hcc -n default
Spec:
Name: test-volumeexpansion-z5hcc NameSpace: default Cluster: test Type: VolumeExpansion
Command:
kbcli cluster volume-expand test --components=cn --volume-claim-template-names=data --storage=21Gi --namespace=default
Last Configuration:
COMPONENT VOLUME-CLAIM-TEMPLATE STORAGE
cn data 20Gi
Status:
Start Time: Sep 26,2023 17:19 UTC+0800
Duration: 6m55s
Status: Running
Progress: 0/1
OBJECT-KEY STATUS DURATION MESSAGE
PVC/data-test-cn-0(data) Processing <Unknown> Start expanding volume: PVC/data-test-cn-0 in Component: cn
Conditions:
LAST-TRANSITION-TIME TYPE REASON STATUS MESSAGE
Sep 26,2023 17:19 UTC+0800 Progressing OpsRequestProgressingStarted True Start to process the OpsRequest: test-volumeexpansion-z5hcc in Cluster: test
Sep 26,2023 17:19 UTC+0800 Validated ValidateOpsRequestPassed True OpsRequest: test-volumeexpansion-z5hcc is validated
Sep 26,2023 17:19 UTC+0800 VolumeExpanding VolumeExpansionStarted True Start to expand the volumes in Cluster: test
Warning Events: <none>
➜ ~ k describe pvc data-test-cn-0
Name: data-test-cn-0
Namespace: default
StorageClass: kb-default-sc
Status: Bound
Volume: pvc-dafc3f90-eb97-4989-8ed0-4e928e84b0b5
Labels: app.kubernetes.io/instance=test
app.kubernetes.io/managed-by=kubeblocks
app.kubernetes.io/name=polardbx
apps.kubeblocks.io/component-name=cn
apps.kubeblocks.io/vct-name=data
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
volume.kubernetes.io/selected-node: gke-yjtest-default-pool-16b6e83c-hsgn
volume.kubernetes.io/storage-provisioner: pd.csi.storage.gke.io
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 20Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: test-cn-0
Conditions:
Type Status LastProbeTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
FileSystemResizePending True Mon, 01 Jan 0001 00:00:00 +0000 Tue, 26 Sep 2023 17:19:42 +0800 Waiting for user to (re-)start a pod to finish file system resize of volume on node.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 40m persistentvolume-controller waiting for first consumer to be created before binding
Normal ExternalProvisioning 40m (x2 over 40m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "pd.csi.storage.gke.io" or manually created by system administrator
Normal Provisioning 40m pd.csi.storage.gke.io_gke-e81b3d5226fe4e088e54-819b-0aa3-vm_8d2cefde-e802-44df-94b2-d1edb676b67a External provisioner is provisioning volume for claim "default/data-test-cn-0"
Normal ProvisioningSucceeded 40m pd.csi.storage.gke.io_gke-e81b3d5226fe4e088e54-819b-0aa3-vm_8d2cefde-e802-44df-94b2-d1edb676b67a Successfully provisioned volume pvc-dafc3f90-eb97-4989-8ed0-4e928e84b0b5
Warning ExternalExpanding 7m40s volume_expand waiting for an external controller to expand this PVC
Normal Resizing 7m39s external-resizer pd.csi.storage.gke.io External resizer is resizing volume pvc-dafc3f90-eb97-4989-8ed0-4e928e84b0b5
Normal FileSystemResizeRequired 7m32s external-resizer pd.csi.storage.gke.io Require file system resize of volume on node
➜ ~ k describe pv pvc-dafc3f90-eb97-4989-8ed0-4e928e84b0b5
Name: pvc-dafc3f90-eb97-4989-8ed0-4e928e84b0b5
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: pd.csi.storage.gke.io
volume.kubernetes.io/provisioner-deletion-secret-name:
volume.kubernetes.io/provisioner-deletion-secret-namespace:
Finalizers: [kubernetes.io/pv-protection external-attacher/pd-csi-storage-gke-io]
StorageClass: kb-default-sc
Status: Bound
Claim: default/data-test-cn-0
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 21Gi
Node Affinity:
Required Terms:
Term 0: topology.gke.io/zone in [us-central1-c]
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: pd.csi.storage.gke.io
FSType: xfs
VolumeHandle: projects/kubeblocks/zones/us-central1-c/disks/pvc-dafc3f90-eb97-4989-8ed0-4e928e84b0b5
ReadOnly: false
VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1695697785164-6118-pd.csi.storage.gke.io
Events: <none>
➜ ~ kbcli report cluster test --with-logs --all-containers reporting cluster information to report-cluster-test-2023-09-26-17-31-59.zip processing manifests OK processing events OK process pod logs OK ➜ ~ kbcli report kubeblocks --with-logs --all-containers --output yaml reporting KubeBlocks information to report-kubeblocks-2023-09-26-17-33-00.zip processing manifests OK processing events OK process pod logs report-kubeblocks-2023-09-26-17-33-00.zip report-cluster-test-2023-09-26-17-31-59.zip
