kubeblocks
kubeblocks copied to clipboard
Published
20 hours ago •
apecloud
apecloud
[BUG]Status kbcli cluster instance is inconsistent with pod status
The status of pod is ContainerCreating while the status of kbcli cluster command is always Running
Steps:
- Create mysql cluster and kbcli addon enable fault-chaos-mesh
kbcli cluster describe mycluster
Name: mycluster Created Time: Jul 11,2023 14:37 UTC+0800
NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY
default apecloud-mysql ac-mysql-8.0.30 Running WipeOut
Endpoints:
COMPONENT MODE INTERNAL EXTERNAL
mysql ReadWrite mycluster-mysql.default.svc.cluster.local:3306 <none>
Topology:
COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME
mysql mycluster-mysql-0 leader Running us-central1-c gke-yjtest-default-pool-50025fd7-vtwk/10.128.0.29 Jul 11,2023 15:49 UTC+0800
mysql mycluster-mysql-1 follower Running us-central1-c gke-yjtest-default-pool-50025fd7-5090/10.128.0.26 Jul 11,2023 15:47 UTC+0800
mysql mycluster-mysql-2 follower Running us-central1-c gke-yjtest-default-pool-50025fd7-zb67/10.128.0.30 Jul 11,2023 15:45 UTC+0800
Resources Allocation:
COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS
mysql false 1 / 1 1Gi / 1Gi data:20Gi standard-rwo
Images:
COMPONENT TYPE IMAGE
mysql mysql registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha9.20230606.gf80d546.9
Data Protection:
AUTO-BACKUP BACKUP-SCHEDULE TYPE BACKUP-TTL LAST-SCHEDULE RECOVERABLE-TIME
Show cluster events: kbcli cluster list-events -n default mycluster
- inject stop node fault to leader node
kbcli fault node stop gke-yjtest-default-pool-50025fd7-vtwk -c=gcp --region=us-central1-c --project=kubeblocks --duration=2m
Secret cloud-key-secret-gcp exists under default namespace.
GCPChaos node-chaos-cn65s created
k get node
NAME STATUS ROLES AGE VERSION
gke-yjtest-default-pool-50025fd7-5090 Ready <none> 90m v1.26.5-gke.1200
gke-yjtest-default-pool-50025fd7-vtwk NotReady <none> 44m v1.26.5-gke.1200
gke-yjtest-default-pool-50025fd7-zb67 Ready <none> 16m v1.26.5-gke.1200
- Status of pod is inconsistent when the node is back to ready that the status of kbcli cluster instance is always Running
k get node
NAME STATUS ROLES AGE VERSION
gke-yjtest-default-pool-50025fd7-5090 Ready <none> 92m v1.26.5-gke.1200
gke-yjtest-default-pool-50025fd7-vtwk Ready <none> 35s v1.26.5-gke.1200
gke-yjtest-default-pool-50025fd7-zb67 Ready <none> 17m v1.26.5-gke.1200
k get pod
NAME READY STATUS RESTARTS AGE
mycluster-mysql-0 0/5 ContainerCreating 5 154m
mycluster-mysql-1 5/5 Running 0 156m
mycluster-mysql-2 5/5 Running 0 158m
kbcli cluster describe mycluster
Name: mycluster Created Time: Jul 11,2023 14:37 UTC+0800
NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY
default apecloud-mysql ac-mysql-8.0.30 Abnormal WipeOut
Endpoints:
COMPONENT MODE INTERNAL EXTERNAL
mysql ReadWrite mycluster-mysql.default.svc.cluster.local:3306 <none>
Topology:
COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME
mysql mycluster-mysql-0 leader Running us-central1-c gke-yjtest-default-pool-50025fd7-vtwk/10.128.0.31 Jul 11,2023 15:49 UTC+0800
mysql mycluster-mysql-1 leader Running us-central1-c gke-yjtest-default-pool-50025fd7-5090/10.128.0.26 Jul 11,2023 15:47 UTC+0800
mysql mycluster-mysql-2 follower Running us-central1-c gke-yjtest-default-pool-50025fd7-zb67/10.128.0.30 Jul 11,2023 15:45 UTC+0800
Resources Allocation:
COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS
mysql false 1 / 1 1Gi / 1Gi data:20Gi standard-rwo
Images:
COMPONENT TYPE IMAGE
mysql mysql registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha9.20230606.gf80d546.9
Data Protection:
AUTO-BACKUP BACKUP-SCHEDULE TYPE BACKUP-TTL LAST-SCHEDULE RECOVERABLE-TIME
Show cluster events: kbcli cluster list-events -n default mycluster
kbcli cluster list-instances mycluster
NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME
mycluster-mysql-0 default mycluster mysql Running leader ReadWrite us-central1-c 1 / 1 1Gi / 1Gi data:20Gi gke-yjtest-default-pool-50025fd7-vtwk/10.128.0.31 Jul 11,2023 15:49 UTC+0800
mycluster-mysql-1 default mycluster mysql Running leader ReadWrite us-central1-c 1 / 1 1Gi / 1Gi data:20Gi gke-yjtest-default-pool-50025fd7-5090/10.128.0.26 Jul 11,2023 15:47 UTC+0800
mycluster-mysql-2 default mycluster mysql Running follower Readonly us-central1-c 1 / 1 1Gi / 1Gi data:20Gi gke-yjtest-default-pool-50025fd7-zb67/10.128.0.30 Jul 11,2023 15:45 UTC+0800
kubectl get cluster mycluster -o yaml
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
annotations:
kubeblocks.io/reconcile: "2023-07-11T10:21:47.195015664Z"
creationTimestamp: "2023-07-11T06:37:13Z"
finalizers:
- cluster.kubeblocks.io/finalizer
generation: 2
labels:
clusterdefinition.kubeblocks.io/name: apecloud-mysql
clusterversion.kubeblocks.io/name: ac-mysql-8.0.30
name: mycluster
namespace: default
resourceVersion: "1183486"
uid: 75bf5c8a-17b2-40a1-9a92-dfe850a12a39
spec:
affinity:
nodeLabels: {}
podAntiAffinity: Preferred
tenancy: SharedNode
topologyKeys: []
clusterDefinitionRef: apecloud-mysql
clusterVersionRef: ac-mysql-8.0.30
componentSpecs:
- componentDefRef: mysql
monitor: true
name: mysql
noCreatePDB: false
replicas: 3
resources:
limits:
cpu: "1"
memory: 1Gi
requests:
cpu: "1"
memory: 1Gi
serviceAccountName: kb-sa-mycluster
volumeClaimTemplates:
- name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
terminationPolicy: WipeOut
tolerations: []
status:
clusterDefGeneration: 2
components:
mysql:
consensusSetStatus:
followers:
- accessMode: Readonly
name: follower
pod: mycluster-mysql-2
leader:
accessMode: ReadWrite
name: leader
pod: mycluster-mysql-1
phase: Abnormal
podsReady: false
conditions:
- lastTransitionTime: "2023-07-11T06:37:26Z"
message: found uncleaned resources, requires manual deletion, check with `kubectl
-n default get pvc,secret,cm -l app.kubernetes.io/instance=mycluster`
reason: UncleanedResources
status: "False"
type: HaltRecovery
- lastTransitionTime: "2023-07-11T06:37:26Z"
message: 'The operator has started the provisioning of Cluster: mycluster'
observedGeneration: 2
reason: PreCheckSucceed
status: "True"
type: ProvisioningStarted
- lastTransitionTime: "2023-07-11T07:49:24Z"
message: Successfully applied for resources
observedGeneration: 2
reason: ApplyResourcesSucceed
status: "True"
type: ApplyResources
- lastTransitionTime: "2023-07-11T10:21:47Z"
message: 'pods are not ready in Components: [mysql], refer to related component
message in Cluster.status.components'
reason: ReplicasNotReady
status: "False"
type: ReplicasReady
- lastTransitionTime: "2023-07-11T10:21:47Z"
message: 'pods are unavailable in Components: [mysql], refer to related component
message in Cluster.status.components'
reason: ComponentsNotReady
status: "False"
type: Ready
observedGeneration: 2
phase: Abnormal
k logs mycluster-mysql-0
Defaulted container "mysql" out of: mysql, metrics, vttablet, kb-checkrole, config-manager
Error from server (BadRequest): container "mysql" in pod "mycluster-mysql-0" is terminated
k describe pod mycluster-mysql-0
Name: mycluster-mysql-0
Namespace: default
Priority: 0
Node: gke-yjtest-default-pool-50025fd7-vtwk/10.128.0.31
Start Time: Tue, 11 Jul 2023 15:49:24 +0800
Labels: app.kubernetes.io/component=mysql
app.kubernetes.io/instance=mycluster
app.kubernetes.io/managed-by=kubeblocks
app.kubernetes.io/name=apecloud-mysql
app.kubernetes.io/version=ac-mysql-8.0.30
apps.kubeblocks.io/component-name=mysql
apps.kubeblocks.io/workload-type=Consensus
controller-revision-hash=mycluster-mysql-dcffcf98f
cs.apps.kubeblocks.io/access-mode=ReadWrite
kubeblocks.io/role=leader
statefulset.kubernetes.io/pod-name=mycluster-mysql-0
Annotations: apps.kubeblocks.io/component-replicas: 3
cs.apps.kubeblocks.io/leader: mycluster-mysql-1
Status: Running
IP:
IPs: <none>
Controlled By: StatefulSet/mycluster-mysql
Containers:
mysql:
Container ID:
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha9.20230606.gf80d546.9
Image ID:
Ports: 3306/TCP, 13306/TCP
Host Ports: 0/TCP, 0/TCP
Command:
/scripts/setup.sh
State: Waiting
Reason: ContainerCreating
Last State: Terminated
Reason: ContainerStatusUnknown
Message: The container could not be located when the pod was deleted. The container used to be Running
Exit Code: 137
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Ready: False
Restart Count: 1
Limits:
cpu: 1
memory: 1Gi
Requests:
cpu: 1
memory: 1Gi
Environment Variables from:
mycluster-mysql-env ConfigMap Optional: false
Environment:
KB_POD_NAME: mycluster-mysql-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: mycluster
KB_COMP_NAME: mysql
KB_CLUSTER_COMP_NAME: mycluster-mysql
KB_CLUSTER_UID_POSTFIX_8: 50a12a39
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
MYSQL_ROOT_HOST: %
MYSQL_ROOT_USER: <set to the key 'username' in secret 'mycluster-conn-credential'> Optional: false
MYSQL_ROOT_PASSWORD: <set to the key 'password' in secret 'mycluster-conn-credential'> Optional: false
MYSQL_DATABASE: mydb
MYSQL_USER: u1
MYSQL_PASSWORD: u1
CLUSTER_ID: 1
CLUSTER_START_INDEX: 1
REPLICATION_USER: replicator
REPLICATION_PASSWORD:
MYSQL_TEMPLATE_CONFIG:
MYSQL_CUSTOM_CONFIG:
MYSQL_DYNAMIC_CONFIG:
KB_EMBEDDED_WESQL: 1
Mounts:
/data/mysql from data (rw)
/etc/annotations from annotations (rw)
/opt/mysql from mysql-config (rw)
/scripts from scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcspw (ro)
metrics:
Container ID:
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1
Image ID:
Port: 9104/TCP
Host Port: 0/TCP
Command:
/bin/agamotto
--config=/opt/agamotto/agamotto-config.yaml
State: Waiting
Reason: ContainerCreating
Last State: Terminated
Reason: ContainerStatusUnknown
Message: The container could not be located when the pod was deleted. The container used to be Running
Exit Code: 137
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Ready: False
Restart Count: 1
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
mycluster-mysql-env ConfigMap Optional: false
Environment:
KB_POD_NAME: mycluster-mysql-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: mycluster
KB_COMP_NAME: mysql
KB_CLUSTER_COMP_NAME: mycluster-mysql
KB_CLUSTER_UID_POSTFIX_8: 50a12a39
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
DB_TYPE: MySQL
ENDPOINT: localhost:3306
MYSQL_USER: <set to the key 'username' in secret 'mycluster-conn-credential'> Optional: false
MYSQL_PASSWORD: <set to the key 'password' in secret 'mycluster-conn-credential'> Optional: false
Mounts:
/data/mysql from data (rw)
/opt/agamotto from agamotto-configuration (rw)
/var/log/kubeblocks from log-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcspw (ro)
vttablet:
Container ID:
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:latest
Image ID:
Ports: 15100/TCP, 16100/TCP, 40000/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/scripts/vttablet.sh
State: Waiting
Reason: ContainerCreating
Last State: Terminated
Reason: ContainerStatusUnknown
Message: The container could not be located when the pod was deleted. The container used to be Running
Exit Code: 137
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Ready: False
Restart Count: 1
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
mycluster-mysql-env ConfigMap Optional: false
Environment:
KB_POD_NAME: mycluster-mysql-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: mycluster
KB_COMP_NAME: mysql
KB_CLUSTER_COMP_NAME: mycluster-mysql
KB_CLUSTER_UID_POSTFIX_8: 50a12a39
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
CELL: zone1
ETCD_SERVER: $(KB_CLUSTER_NAME)-etcd-headless
ETCD_PORT: 2379
TOPOLOGY_FLAGS: --topo_implementation etcd2 --topo_global_server_address $(ETCD_SERVER):$(ETCD_PORT) --topo_global_root /vitess/global
VTTABLET_PORT: 15100
VTTABLET_GRPC_PORT: 16100
VTCTLD_HOST: $(KB_CLUSTER_NAME)-vtctld-headless
VTCTLD_WEB_PORT: 15000
MYSQL_ROOT_USER: <set to the key 'username' in secret 'mycluster-conn-credential'> Optional: false
MYSQL_ROOT_PASSWORD: <set to the key 'password' in secret 'mycluster-conn-credential'> Optional: false
Mounts:
/conf from mysql-scale-config (rw)
/scripts from scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcspw (ro)
kb-checkrole:
Container ID:
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.34
Image ID:
Ports: 3501/TCP, 50001/TCP
Host Ports: 0/TCP, 0/TCP
Command:
probe
--app-id
batch-sdk
--dapr-http-port
3501
--dapr-grpc-port
50001
--log-level
info
--config
/config/probe/config.yaml
--components-path
/config/probe/components
State: Waiting
Reason: ContainerCreating
Last State: Terminated
Reason: ContainerStatusUnknown
Message: The container could not be located when the pod was deleted. The container used to be Running
Exit Code: 137
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Ready: False
Restart Count: 1
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Readiness: http-get http://:3501/v1.0/bindings/mysql%3Foperation=checkRole delay=0s timeout=1s period=1s #success=1 #failure=2
Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
mycluster-mysql-env ConfigMap Optional: false
Environment:
KB_POD_NAME: mycluster-mysql-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: mycluster
KB_COMP_NAME: mysql
KB_CLUSTER_COMP_NAME: mycluster-mysql
KB_CLUSTER_UID_POSTFIX_8: 50a12a39
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
KB_SERVICE_USER: <set to the key 'username' in secret 'mycluster-conn-credential'> Optional: false
KB_SERVICE_PASSWORD: <set to the key 'password' in secret 'mycluster-conn-credential'> Optional: false
KB_SERVICE_PORT: 3306
KB_SERVICE_ROLES: {"follower":"Readonly","leader":"ReadWrite","learner":"Readonly"}
KB_SERVICE_CHARACTER_TYPE: mysql
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcspw (ro)
config-manager:
Container ID:
Image: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.34
Image ID:
Port: <none>
Host Port: <none>
Command:
env
Args:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH)
/bin/reloader
--log-level
info
--volume-dir
/conf
--operator-update-enable
--tcp
9901
--config
/opt/config-manager/config-manager.yaml
State: Waiting
Reason: ContainerCreating
Last State: Terminated
Reason: ContainerStatusUnknown
Message: The container could not be located when the pod was deleted. The container used to be Running
Exit Code: 137
Started: Mon, 01 Jan 0001 00:00:00 +0000
Finished: Mon, 01 Jan 0001 00:00:00 +0000
Ready: False
Restart Count: 1
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
mycluster-mysql-env ConfigMap Optional: false
Environment:
KB_POD_NAME: mycluster-mysql-0 (v1:metadata.name)
KB_POD_UID: (v1:metadata.uid)
KB_NAMESPACE: default (v1:metadata.namespace)
KB_SA_NAME: (v1:spec.serviceAccountName)
KB_NODENAME: (v1:spec.nodeName)
KB_HOST_IP: (v1:status.hostIP)
KB_POD_IP: (v1:status.podIP)
KB_POD_IPS: (v1:status.podIPs)
KB_HOSTIP: (v1:status.hostIP)
KB_PODIP: (v1:status.podIP)
KB_PODIPS: (v1:status.podIPs)
KB_CLUSTER_NAME: mycluster
KB_COMP_NAME: mysql
KB_CLUSTER_COMP_NAME: mycluster-mysql
KB_CLUSTER_UID_POSTFIX_8: 50a12a39
KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
CONFIG_MANAGER_POD_IP: (v1:status.podIP)
DB_TYPE: mysql
MYSQL_USER: <set to the key 'username' in secret 'mycluster-conn-credential'> Optional: false
MYSQL_PASSWORD: <set to the key 'password' in secret 'mycluster-conn-credential'> Optional: false
DATA_SOURCE_NAME: $(MYSQL_USER):$(MYSQL_PASSWORD)@(localhost:3306)/
TOOLS_PATH: /opt/kb-tools/reload/mysql-consensusset-config:/opt/config-manager
Mounts:
/conf from mysql-scale-config (rw)
/opt/config-manager from config-manager-config (rw)
/opt/kb-tools/reload/mysql-consensusset-config from cm-script-mysql-consensusset-config (rw)
/opt/mysql from mysql-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcspw (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-mycluster-mysql-0
ReadOnly: false
log-data:
Type: HostPath (bare host directory volume)
Path: /var/log/kubeblocks
HostPathType: DirectoryOrCreate
annotations:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.annotations['cs.apps.kubeblocks.io/leader'] -> leader
metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas
agamotto-configuration:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: mycluster-mysql-agamotto-configuration
Optional: false
scripts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: mycluster-mysql-apecloud-mysql-scripts
Optional: false
mysql-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: mycluster-mysql-mysql-consensusset-config
Optional: false
mysql-scale-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: mycluster-mysql-mysql-scale-config
Optional: false
cm-script-mysql-consensusset-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: sidecar-mysql-reload-script-mycluster
Optional: false
config-manager-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: sidecar-mycluster-mysql-config-manager-config
Optional: false
kube-api-access-fcspw:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: kb-data=true:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal checkRole 59m sqlchannel {"event":"Success","operation":"checkRole","originalRole":"Follower","role":"Leader"}
Warning Unhealthy 51m kubelet Readiness probe failed: Get "http://10.104.2.7:3501/v1.0/bindings/mysql?operation=checkRole": dial tcp 10.104.2.7:3501: connect: connection refused
Warning FailedMount 47m (x8 over 48m) kubelet MountVolume.NodeAffinity check failed for volume "pvc-f5fcacf2-56ea-48c5-9963-32d149029ddd" : no matching NodeSelectorTerms
Warning FailedMount 46m kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[log-data config-manager-config mysql-scale-config annotations kube-api-access-fcspw agamotto-configuration cm-script-mysql-consensusset-config data mysql-config scripts]: timed out waiting for the condition
Normal Pulling 46m kubelet Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha9.20230606.gf80d546.9"
Normal Created 36m kubelet Created container mysql
Normal Started 36m kubelet Started container mysql
Normal Pulling 36m kubelet Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1"
Normal Pulled 36m kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha9.20230606.gf80d546.9" in 9m49.275940754s (9m49.275949118s including waiting)
Normal Pulled 36m kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1" in 24.631034884s (24.631046851s including waiting)
Normal Created 36m kubelet Created container metrics
Normal Pulling 36m kubelet Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:latest"
Normal Started 36m kubelet Started container metrics
Normal Pulled 35m kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:latest" in 1m17.915772496s (1m17.915787768s including waiting)
Normal Created 35m kubelet Created container vttablet
Normal Started 34m kubelet Started container vttablet
Normal Pulling 34m kubelet Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.34"
Normal Pulled 34m kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.34" in 28.096796489s (28.096809048s including waiting)
Normal Created 34m kubelet Created container kb-checkrole
Normal Started 34m kubelet Started container kb-checkrole
Normal Pulled 34m kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.34" already present on machine
Normal Created 34m kubelet Created container config-manager
Normal checkRole 34m sqlchannel {"event":"Success","operation":"checkRole","originalRole":"","role":"Follower"}
Normal checkRole 21m sqlchannel {"event":"Success","operation":"checkRole","originalRole":"Follower","role":"Leader"}
Warning Unhealthy 5m11s kubelet Readiness probe failed: Get "http://10.104.2.7:3501/v1.0/bindings/mysql?operation=checkRole": dial tcp 10.104.2.7:3501: connect: connection refused
Warning NodeNotReady 4m33s (x3 over 72m) node-controller Node is not ready
Warning NetworkNotReady 2m48s kubelet network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Warning FailedMount 2m46s (x3 over 2m48s) kubelet MountVolume.SetUp failed for volume "agamotto-configuration" : object "default"/"mycluster-mysql-agamotto-configuration" not registered
Warning FailedMount 2m46s (x3 over 2m48s) kubelet MountVolume.SetUp failed for volume "mysql-scale-config" : object "default"/"mycluster-mysql-mysql-scale-config" not registered
Warning FailedMount 2m46s (x3 over 2m48s) kubelet MountVolume.SetUp failed for volume "scripts" : object "default"/"mycluster-mysql-apecloud-mysql-scripts" not registered
Warning FailedMount 2m46s (x3 over 2m48s) kubelet MountVolume.SetUp failed for volume "kube-api-access-fcspw" : object "default"/"kube-root-ca.crt" not registered
Warning FailedMount 2m46s (x3 over 2m48s) kubelet MountVolume.SetUp failed for volume "mysql-config" : object "default"/"mycluster-mysql-mysql-consensusset-config" not registered
Warning FailedMount 2m46s (x3 over 2m48s) kubelet MountVolume.SetUp failed for volume "cm-script-mysql-consensusset-config" : object "default"/"sidecar-mysql-reload-script-mycluster" not registered
Warning FailedMount 2m46s (x3 over 2m48s) kubelet MountVolume.SetUp failed for volume "config-manager-config" : object "default"/"sidecar-mycluster-mysql-config-manager-config" not registered
Warning FailedMount 2m35s (x3 over 2m38s) kubelet MountVolume.MountDevice failed for volume "pvc-f5fcacf2-56ea-48c5-9963-32d149029ddd" : rpc error: code = Internal desc = Error when getting device path: rpc error: code = Internal desc = error verifying GCE PD ("pvc-f5fcacf2-56ea-48c5-9963-32d149029ddd") is attached: failed to find and re-link disk pvc-f5fcacf2-56ea-48c5-9963-32d149029ddd with udevadm after retrying for 3s: for disk pvc-f5fcacf2-56ea-48c5-9963-32d149029ddd failed to trigger udevadm fix of non existent device path: udevadm --trigger requested to fix disk pvc-f5fcacf2-56ea-48c5-9963-32d149029ddd but no such disk was found in device path map[/dev/sda:persistent-disk-0 /dev/sda1:persistent-disk-0 /dev/sda10:persistent-disk-0 /dev/sda11:persistent-disk-0 /dev/sda12:persistent-disk-0 /dev/sda2:persistent-disk-0 /dev/sda3:persistent-disk-0 /dev/sda4:persistent-disk-0 /dev/sda5:persistent-disk-0 /dev/sda6:persistent-disk-0 /dev/sda7:persistent-disk-0 /dev/sda8:persistent-disk-0 /dev/sda9:persistent-disk-0]
