kubeblocks
kubeblocks copied to clipboard
Published
20 hours ago •
apecloud
apecloud
[BUG] pika cluster hscale in component pika-group ops is always running
Describe the bug
kbcli version
Kubernetes: v1.27.9
KubeBlocks: 0.9.0-beta.27
kbcli: 0.9.0-beta.23
To Reproduce Steps to reproduce the behavior:
- create cluster
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
name: release-name-pika-cluster
namespace: default
labels:
helm.sh/chart: pika-cluster-0.7.1-beta.1
app.kubernetes.io/name: pika-cluster
app.kubernetes.io/instance: release-name
app.kubernetes.io/version: "3.5.3"
app.kubernetes.io/managed-by: Helm
spec:
clusterDefinitionRef: pika # ref clusterDefinition.name
clusterVersionRef: pika-3.5.3 # ref clusterVersion.name
terminationPolicy: Delete
affinity:
topologyKeys:
- kubernetes.io/hostname
## define pika group with shardingSpecs API which is supported in KubeBlocks v0.8.2
componentSpecs:
- name: pika-group-1 # user-defined
componentDefRef: pika-group # ref clusterDefinition.componentDefs[x].name
monitor: false
enabledLogs: null
replicas: 2
serviceAccountName: kb-release-name-pika-cluster
switchPolicy:
type: Noop
resources:
limits:
cpu: "500m"
memory: "3Gi"
requests:
cpu: "500m"
memory: "1Gi"
volumeClaimTemplates:
- name: data # ref clusterdefinition components.containers.volumeMounts.name
spec:
storageClassName:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- name: pika-group-2 # user-defined
componentDefRef: pika-group # ref clusterDefinition.componentDefs[x].name
monitor: false
enabledLogs: null
replicas: 2
serviceAccountName: kb-release-name-pika-cluster
switchPolicy:
type: Noop
resources:
limits:
cpu: "500m"
memory: "3Gi"
requests:
cpu: "500m"
memory: "1Gi"
volumeClaimTemplates:
- name: data # ref clusterdefinition components.containers.volumeMounts.name
spec:
storageClassName:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- name: etcd # user-defined
componentDefRef: etcd # ref clusterDefinition.componentDefs[x].name
monitor: false
replicas: 3
resources:
limits:
cpu: "500m"
memory: "3Gi"
requests:
cpu: "500m"
memory: "1Gi"
volumeClaimTemplates:
- name: data # ref clusterdefinition components.containers.volumeMounts.name
spec:
storageClassName:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
- name: pika-exporter
componentDefRef: pika-exporter # ref clusterDefinition.componentDefs[x].name
monitor: false
replicas: 1
resources:
limits:
cpu: "500m"
memory: "3Gi"
requests:
cpu: "500m"
memory: "1Gi"
- name: codis-proxy
componentDefRef: codis-proxy # ref clusterDefinition.componentDefs[x].name
replicas: 2
resources:
limits:
cpu: "500m"
memory: "3Gi"
requests:
cpu: "500m"
memory: "1Gi"
- name: codis-fe
componentDefRef: codis-fe # ref clusterDefinition.componentDefs[x].name
replicas: 1
resources:
limits:
cpu: "500m"
memory: "3Gi"
requests:
cpu: "500m"
memory: "1Gi"
- name: codis-dashboard
componentDefRef: codis-dashboard # ref clusterDefinition.componentDefs[x].name
replicas: 1
resources:
limits:
cpu: "500m"
memory: "3Gi"
requests:
cpu: "500m"
memory: "1Gi"
- hscale out pika-group
kbcli cluster hscale pikacluster-pika-cluster --components pika-group-1 --replicas 3 --auto-approve
- hscale in pika-group
kbcli cluster hscale pikacluster-pika-cluster --components pika-group-1 --replicas 2 --auto-approve
➜ ~ kubectl get cluster pikacluster-pika-cluster
NAME CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS AGE
pikacluster-pika-cluster pika pika-3.5.3 Delete Running 17m
➜ ~
➜ ~ kubectl get pod -l app.kubernetes.io/instance=pikacluster-pika-cluster
NAME READY STATUS RESTARTS AGE
pikacluster-pika-cluster-codis-dashboard-0 1/1 Running 0 18m
pikacluster-pika-cluster-codis-fe-0 1/1 Running 0 18m
pikacluster-pika-cluster-codis-proxy-0 1/1 Running 0 18m
pikacluster-pika-cluster-codis-proxy-1 1/1 Running 0 15m
pikacluster-pika-cluster-etcd-0 1/1 Running 0 18m
pikacluster-pika-cluster-etcd-1 1/1 Running 0 16m
pikacluster-pika-cluster-etcd-2 1/1 Running 0 16m
pikacluster-pika-cluster-pika-exporter-0 1/1 Running 0 18m
pikacluster-pika-cluster-pika-group-1-0 3/3 Running 0 18m
pikacluster-pika-cluster-pika-group-1-1 3/3 Running 0 16m
pikacluster-pika-cluster-pika-group-1-2 3/3 Running 0 14m
pikacluster-pika-cluster-pika-group-2-0 3/3 Running 0 18m
pikacluster-pika-cluster-pika-group-2-1 3/3 Running 0 16m
➜ ~
- See error
kubectl get ops
NAME TYPE CLUSTER STATUS PROGRESS AGE
pikacluster-pika-cluster-horizontalscaling-4wnkm HorizontalScaling pikacluster-pika-cluster Succeed 1/1 10m
pikacluster-pika-cluster-horizontalscaling-xmlfw HorizontalScaling pikacluster-pika-cluster Running 0/1 6m22s
describe ops
kubectl describe ops pikacluster-pika-cluster-horizontalscaling-xmlfw
Name: pikacluster-pika-cluster-horizontalscaling-xmlfw
Namespace: default
Labels: app.kubernetes.io/instance=pikacluster-pika-cluster
app.kubernetes.io/managed-by=kubeblocks
ops.kubeblocks.io/ops-type=HorizontalScaling
Annotations: <none>
API Version: apps.kubeblocks.io/v1alpha1
Kind: OpsRequest
Metadata:
Creation Timestamp: 2024-05-28T04:08:47Z
Finalizers:
opsrequest.kubeblocks.io/finalizer
Generate Name: pikacluster-pika-cluster-horizontalscaling-
Generation: 2
Managed Fields:
API Version: apps.kubeblocks.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:generateName:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/managed-by:
f:spec:
.:
f:clusterName:
f:horizontalScaling:
.:
k:{"componentName":"pika-group-1"}:
.:
f:componentName:
f:replicas:
f:preConditionDeadlineSeconds:
f:type:
Manager: kbcli
Operation: Update
Time: 2024-05-28T04:08:47Z
API Version: apps.kubeblocks.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"opsrequest.kubeblocks.io/finalizer":
f:labels:
f:ops.kubeblocks.io/ops-type:
f:ownerReferences:
.:
k:{"uid":"95e283a4-8754-432f-b7de-268cf5144ee5"}:
Manager: manager
Operation: Update
Time: 2024-05-28T04:08:47Z
API Version: apps.kubeblocks.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:clusterGeneration:
f:components:
.:
f:pika-group-1:
.:
f:phase:
f:progressDetails:
f:conditions:
.:
k:{"type":"HorizontalScaling"}:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
k:{"type":"Validated"}:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
k:{"type":"WaitForProgressing"}:
.:
f:lastTransitionTime:
f:message:
f:reason:
f:status:
f:type:
f:lastConfiguration:
.:
f:components:
.:
f:pika-group-1:
.:
f:replicas:
f:phase:
f:progress:
f:startTimestamp:
Manager: manager
Operation: Update
Subresource: status
Time: 2024-05-28T04:08:47Z
Owner References:
API Version: apps.kubeblocks.io/v1alpha1
Kind: Cluster
Name: pikacluster-pika-cluster
UID: 95e283a4-8754-432f-b7de-268cf5144ee5
Resource Version: 6609476
UID: 5678c587-da14-434e-a3de-64b624a24b43
Spec:
Cluster Name: pikacluster-pika-cluster
Horizontal Scaling:
Component Name: pika-group-1
Replicas: 2
Pre Condition Deadline Seconds: 0
Type: HorizontalScaling
Status:
Cluster Generation: 4
Components:
pika-group-1:
Phase: Running
Progress Details:
Group: pika-group-1
Message: wait to delete pod: Pod/pikacluster-pika-cluster-pika-group-1-2 in Component: pika-group-1
Object Key: Pod/pikacluster-pika-cluster-pika-group-1-2
Status: Pending
Conditions:
Last Transition Time: 2024-05-28T04:08:47Z
Message: wait for the controller to process the OpsRequest: pikacluster-pika-cluster-horizontalscaling-xmlfw in Cluster: pikacluster-pika-cluster
Reason: WaitForProgressing
Status: True
Type: WaitForProgressing
Last Transition Time: 2024-05-28T04:08:47Z
Message: OpsRequest: pikacluster-pika-cluster-horizontalscaling-xmlfw is validated
Reason: ValidateOpsRequestPassed
Status: True
Type: Validated
Last Transition Time: 2024-05-28T04:08:47Z
Message: Start to horizontal scale replicas in Cluster: pikacluster-pika-cluster
Reason: HorizontalScalingStarted
Status: True
Type: HorizontalScaling
Last Configuration:
Components:
pika-group-1:
Replicas: 3
Phase: Running
Progress: 0/1
Start Timestamp: 2024-05-28T04:08:47Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForProgressing 9m6s ops-request-controller wait for the controller to process the OpsRequest: pikacluster-pika-cluster-horizontalscaling-xmlfw in Cluster: pikacluster-pika-cluster
Normal ValidateOpsRequestPassed 9m6s ops-request-controller OpsRequest: pikacluster-pika-cluster-horizontalscaling-xmlfw is validated
Normal HorizontalScalingStarted 9m6s ops-request-controller Start to horizontal scale replicas in Cluster: pikacluster-pika-cluster
➜ ~
describe cluster
kubectl describe cluster pikacluster-pika-cluster
Name: pikacluster-pika-cluster
Namespace: default
Labels: app.kubernetes.io/instance=pikacluster
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=pika-cluster
app.kubernetes.io/version=3.5.3
clusterdefinition.kubeblocks.io/name=pika
clusterversion.kubeblocks.io/name=pika-3.5.3
helm.sh/chart=pika-cluster-0.7.1-beta.1
Annotations: kubeblocks.io/ops-request: [{"name":"pikacluster-pika-cluster-horizontalscaling-xmlfw","type":"HorizontalScaling"}]
meta.helm.sh/release-name: pikacluster
meta.helm.sh/release-namespace: default
API Version: apps.kubeblocks.io/v1alpha1
Kind: Cluster
Metadata:
Creation Timestamp: 2024-05-28T04:00:45Z
Finalizers:
cluster.kubeblocks.io/finalizer
Generation: 4
Managed Fields:
API Version: apps.kubeblocks.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:meta.helm.sh/release-name:
f:meta.helm.sh/release-namespace:
f:labels:
.:
f:app.kubernetes.io/instance:
f:app.kubernetes.io/managed-by:
f:app.kubernetes.io/name:
f:app.kubernetes.io/version:
f:helm.sh/chart:
f:spec:
.:
f:affinity:
.:
f:podAntiAffinity:
f:tenancy:
f:topologyKeys:
.:
v:"kubernetes.io/hostname":
f:clusterDefinitionRef:
f:clusterVersionRef:
f:terminationPolicy:
Manager: helm
Operation: Update
Time: 2024-05-28T04:00:45Z
API Version: apps.kubeblocks.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:kubeblocks.io/ops-request:
f:finalizers:
.:
v:"cluster.kubeblocks.io/finalizer":
f:labels:
f:clusterdefinition.kubeblocks.io/name:
f:clusterversion.kubeblocks.io/name:
f:spec:
f:componentSpecs:
f:resources:
.:
f:cpu:
f:memory:
f:storage:
.:
f:size:
Manager: manager
Operation: Update
Time: 2024-05-28T04:08:47Z
API Version: apps.kubeblocks.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:clusterDefGeneration:
f:components:
.:
f:codis-dashboard:
.:
f:phase:
f:podsReady:
f:podsReadyTime:
f:codis-fe:
.:
f:phase:
f:podsReady:
f:podsReadyTime:
f:codis-proxy:
.:
f:phase:
f:podsReady:
f:podsReadyTime:
f:etcd:
.:
f:phase:
f:podsReady:
f:podsReadyTime:
f:pika-exporter:
.:
f:phase:
f:podsReady:
f:podsReadyTime:
f:pika-group-1:
.:
f:phase:
f:podsReady:
f:podsReadyTime:
f:pika-group-2:
.:
f:phase:
f:podsReady:
f:podsReadyTime:
f:conditions:
f:observedGeneration:
f:phase:
Manager: manager
Operation: Update
Subresource: status
Time: 2024-05-28T04:08:52Z
Resource Version: 6609547
UID: 95e283a4-8754-432f-b7de-268cf5144ee5
Spec:
Affinity:
Pod Anti Affinity: Preferred
Tenancy: SharedNode
Topology Keys:
kubernetes.io/hostname
Cluster Definition Ref: pika
Cluster Version Ref: pika-3.5.3
Component Specs:
Component Def Ref: pika-group
Monitor: false
Name: pika-group-1
Replicas: 2
Resources:
Limits:
Cpu: 500m
Memory: 3Gi
Requests:
Cpu: 500m
Memory: 1Gi
Service Account Name: kb-pikacluster-pika-cluster
Switch Policy:
Type: Noop
Volume Claim Templates:
Name: data
Spec:
Access Modes:
ReadWriteOnce
Resources:
Requests:
Storage: 10Gi
Component Def Ref: pika-group
Monitor: false
Name: pika-group-2
Replicas: 2
Resources:
Limits:
Cpu: 500m
Memory: 3Gi
Requests:
Cpu: 500m
Memory: 1Gi
Service Account Name: kb-pikacluster-pika-cluster
Switch Policy:
Type: Noop
Volume Claim Templates:
Name: data
Spec:
Access Modes:
ReadWriteOnce
Resources:
Requests:
Storage: 10Gi
Component Def Ref: etcd
Monitor: false
Name: etcd
Replicas: 3
Resources:
Limits:
Cpu: 500m
Memory: 3Gi
Requests:
Cpu: 500m
Memory: 1Gi
Volume Claim Templates:
Name: data
Spec:
Access Modes:
ReadWriteOnce
Resources:
Requests:
Storage: 1Gi
Component Def Ref: pika-exporter
Monitor: false
Name: pika-exporter
Replicas: 1
Resources:
Limits:
Cpu: 500m
Memory: 3Gi
Requests:
Cpu: 500m
Memory: 1Gi
Component Def Ref: codis-proxy
Name: codis-proxy
Replicas: 2
Resources:
Limits:
Cpu: 500m
Memory: 3Gi
Requests:
Cpu: 500m
Memory: 1Gi
Component Def Ref: codis-fe
Name: codis-fe
Replicas: 1
Resources:
Limits:
Cpu: 500m
Memory: 3Gi
Requests:
Cpu: 500m
Memory: 1Gi
Component Def Ref: codis-dashboard
Name: codis-dashboard
Replicas: 1
Resources:
Limits:
Cpu: 500m
Memory: 3Gi
Requests:
Cpu: 500m
Memory: 1Gi
Resources:
Cpu: 0
Memory: 0
Storage:
Size: 0
Termination Policy: Delete
Status:
Cluster Def Generation: 3
Components:
Codis - Dashboard:
Phase: Running
Pods Ready: true
Pods Ready Time: 2024-05-28T04:08:49Z
Codis - Fe:
Phase: Running
Pods Ready: true
Pods Ready Time: 2024-05-28T04:08:49Z
Codis - Proxy:
Phase: Running
Pods Ready: true
Pods Ready Time: 2024-05-28T04:08:49Z
Etcd:
Phase: Running
Pods Ready: true
Pods Ready Time: 2024-05-28T04:08:47Z
Pika - Exporter:
Phase: Running
Pods Ready: true
Pods Ready Time: 2024-05-28T04:08:52Z
pika-group-1:
Phase: Running
Pods Ready: true
Pods Ready Time: 2024-05-28T04:05:06Z
pika-group-2:
Phase: Running
Pods Ready: true
Pods Ready Time: 2024-05-28T04:08:51Z
Conditions:
Last Transition Time: 2024-05-28T04:00:45Z
Message: The operator has started the provisioning of Cluster: pikacluster-pika-cluster
Observed Generation: 4
Reason: PreCheckSucceed
Status: True
Type: ProvisioningStarted
Last Transition Time: 2024-05-28T04:04:34Z
Message: Successfully applied for resources
Observed Generation: 4
Reason: ApplyResourcesSucceed
Status: True
Type: ApplyResources
Last Transition Time: 2024-05-28T04:08:52Z
Message: all pods of components are ready, waiting for the probe detection successful
Reason: AllReplicasReady
Status: True
Type: ReplicasReady
Last Transition Time: 2024-05-28T04:08:52Z
Message: Cluster: pikacluster-pika-cluster is ready, current phase is Running
Reason: ClusterReady
Status: True
Type: Ready
Observed Generation: 4
Phase: Running
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ApplyResourcesSucceed 18m cluster-controller Successfully applied for resources
Normal ComponentPhaseTransition 18m (x7 over 18m) cluster-controller component is Creating
Warning ReplicasNotReady 16m cluster-controller pods are not ready in Components: [codis-dashboard codis-fe codis-proxy pika-exporter pika-group-1 pika-group-2], refer to related component message in Cluster.status.components
Warning ComponentsNotReady 16m cluster-controller pods are unavailable in Components: [codis-dashboard codis-fe codis-proxy pika-exporter pika-group-1 pika-group-2], refer to related component message in Cluster.status.components
Warning ReplicasNotReady 16m cluster-controller pods are not ready in Components: [codis-dashboard codis-proxy pika-exporter pika-group-1 pika-group-2], refer to related component message in Cluster.status.components
Warning ComponentsNotReady 16m cluster-controller pods are unavailable in Components: [codis-dashboard codis-proxy pika-exporter pika-group-1 pika-group-2], refer to related component message in Cluster.status.components
Warning ComponentsNotReady 16m cluster-controller pods are unavailable in Components: [codis-dashboard pika-exporter pika-group-1 pika-group-2], refer to related component message in Cluster.status.components
Warning ReplicasNotReady 16m cluster-controller pods are not ready in Components: [codis-dashboard pika-exporter pika-group-1 pika-group-2], refer to related component message in Cluster.status.components
Warning ReplicasNotReady 16m cluster-controller pods are not ready in Components: [pika-exporter pika-group-1 pika-group-2], refer to related component message in Cluster.status.components
Warning ComponentsNotReady 16m cluster-controller pods are unavailable in Components: [pika-exporter pika-group-1 pika-group-2], refer to related component message in Cluster.status.components
Warning ComponentsNotReady 16m cluster-controller pods are unavailable in Components: [pika-exporter pika-group-1], refer to related component message in Cluster.status.components
Normal ComponentPhaseTransition 16m (x6 over 16m) cluster-controller component is Running
Warning ReplicasNotReady 16m cluster-controller pods are not ready in Components: [pika-exporter pika-group-1], refer to related component message in Cluster.status.components
Normal HorizontalScale 15m (x2 over 15m) component-controller start horizontal scale component pika-group-1 of cluster pikacluster-pika-cluster from 2 to 3
Normal PreCheckSucceed 10m (x4 over 18m) cluster-controller The operator has started the provisioning of Cluster: pikacluster-pika-cluster
➜ ~
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
Additional context Add any other context about the problem here.
