kubeblocks
kubeblocks copied to clipboard
apecloud
[BUG] qdrant upgrade to v1 pod crash
Describe the bug A clear and concise description of what the bug is.
kbcli version
Kubernetes: v1.30.4-vke.4
KubeBlocks: 1.0.0-beta.41,0.9.4-beta.10
kbcli: 1.0.0-beta.20
To Reproduce Steps to reproduce the behavior:
- install qdrant addon
helm list -n kb-system
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
kb-addon-qdrant kb-system 2 2025-04-02 14:24:59.566767 +0800 CST deployed qdrant-0.9.1 1.10.0
- create cluster
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
name: qdrant-cluster
namespace: default
spec:
clusterDefinitionRef: qdrant
topology: cluster
terminationPolicy: WipeOut
componentSpecs:
- name: qdrant
serviceVersion: 1.5.0
replicas: 2
serviceAccountName: kb-qdrant-cluster
resources:
requests:
cpu: 100m
memory: 0.5Gi
limits:
cpu: 100m
memory: 0.5Gi
volumeClaimTemplates:
- name: data
spec:
storageClassName:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
- install kb 1.0.0-beta.41
- install qdrant kakfa 1.0.0-alpha.0
helm list -n kb-system-new
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
kb-addon-qdrant kb-system-new 1 2025-04-03 10:28:16.936306 +0800 CST deployed qdrant-1.0.0-alpha.0 1.10.0
- upgrade cluster to v1
helm list -n kb-system-new
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
kb-addon-qdrant kb-system-new 1 2025-04-03 10:28:16.936306 +0800 CST deployed qdrant-1.0.0-alpha.0 1.10.0
- see error
➜ ~ kubectl get cluster qdrant-cluster
NAME CLUSTER-DEFINITION TERMINATION-POLICY STATUS AGE
qdrant-cluster qdrant WipeOut Updating 21h
➜ ~
➜ ~ kubectl get pod -l app.kubernetes.io/instance=qdrant-cluster
NAME READY STATUS RESTARTS AGE
qdrant-cluster-qdrant-0 2/2 Running 0 21h
qdrant-cluster-qdrant-1 1/2 Running 17 (25s ago) 85m
logs errror pod
kubectl logs qdrant-cluster-qdrant-1 --previous
Defaulted container "qdrant" out of: qdrant, kbagent, qdrant-tools (init), init-kbagent (init), kbagent-worker (init)
BOOTSTRAP_HOSTNAME: .default.svc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: .default.svc
INFO: wait for bootstrap node starting...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: .default.svc
INFO: wait for bootstrap node starting...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: .default.svc
...
INFO: wait for bootstrap node starting...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: .default.svc
INFO: wait for bootstrap node starting...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: .default.svc
INFO: wait for bootstrap node starting...
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
Additional context Add any other context about the problem here.
kbcli version
Kubernetes: v1.30.4-vke.5
KubeBlocks: 1.0.0-beta.47,0.9.4-beta.20
kbcli: 1.0.0-beta.21
still error
echo yes|kbcli cluster upgrade-to-v1 qdrant-cluster
┌──────────────────────────────────────────────────────────────┐ ┌────────────────────────────────────────────────────────┐
│apiVersion: apps.kubeblocks.io/v1alpha1 │ │apiVersion: apps.kubeblocks.io/v1 │
│kind: Cluster │ │kind: Cluster │
│metadata: │ │metadata: │
│ annotations: │ │ annotations: │
│ kubeblocks.io/crd-api-version: apps.kubeblocks.io/v1alpha1│ │ kubeblocks.io/crd-api-version: apps.kubeblocks.io/v1│
│ creationTimestamp: "2025-04-28T07:23:40Z" │ │ creationTimestamp: "2025-04-28T07:23:40Z" │
│ finalizers: │ │ finalizers: │
│ - cluster.kubeblocks.io/finalizer │ │ - cluster.kubeblocks.io/finalizer │
│ generation: 2 │ │ generation: 2 │
│ labels: │ │ labels: │
│ clusterdefinition.kubeblocks.io/name: qdrant │ │ clusterdefinition.kubeblocks.io/name: qdrant │
│ clusterversion.kubeblocks.io/name: "" │ │ clusterversion.kubeblocks.io/name: "" │
│ name: qdrant-cluster │ │ name: qdrant-cluster │
│ namespace: default │ │ namespace: default │
│ resourceVersion: "36304" │ │ resourceVersion: "36304" │
│ uid: 9ef5913e-c942-423f-8d65-c24f96035f2a │ │ uid: 9ef5913e-c942-423f-8d65-c24f96035f2a │
│spec: │ │spec: │
│ clusterDefinitionRef: qdrant │ │ clusterDef: qdrant │
│ componentSpecs: │ │ componentSpecs: │
│ - componentDef: qdrant │ │ - componentDef: qdrant-1.0.0-alpha.0 │
│ name: qdrant │ │ name: qdrant │
│ replicas: 2 │ │ replicas: 2 │
│ resources: │ │ resources: │
│ limits: │ │ limits: │
│ cpu: 100m │ │ cpu: 100m │
│ memory: 512Mi │ │ memory: 512Mi │
│ requests: │ │ requests: │
│ cpu: 100m │ │ cpu: 100m │
│ memory: 512Mi │ │ memory: 512Mi │
│ serviceAccountName: kb-qdrant-cluster │ │ serviceVersion: 1.5.0 │
│ serviceVersion: 1.5.0 │ │ volumeClaimTemplates: │
│ volumeClaimTemplates: │ │ - name: data │
│ - name: data │ │ spec: │
│ spec: │ │ accessModes: │
│ accessModes: │ │ - ReadWriteOnce │
│ - ReadWriteOnce │ │ resources: │
│ resources: │ │ requests: │
│ requests: │ │ storage: 20Gi │
│ storage: 20Gi │ │ terminationPolicy: WipeOut │
│ resources: │ │ topology: cluster │
│ cpu: "0" │ │status: {} │
│ memory: "0" │ │ │
│ storage: │ └────────────────────────────────────────────────────────┘
│ size: "0" │
│ terminationPolicy: WipeOut │
│ topology: cluster │
│status: {} │
│ │
└──────────────────────────────────────────────────────────────┘
Cluster qdrant-cluster will be converted to v1 with output as yaml.
Please type 'Yes/yes' to confirm your operation: yes
qdrant-cluster-qdrant-qdrant
Cluster qdrant-cluster has converted successfully, you can view the spec:
kubectl get clusters.apps.kubeblocks.io qdrant-cluster -n default -oyaml
kubectl get clusters.apps.kubeblocks.io qdrant-cluster -n default -oyaml
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
annotations:
kubeblocks.io/crd-api-version: apps.kubeblocks.io/v1
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":{"annotations":{},"name":"qdrant-cluster","namespace":"default"},"spec":{"clusterDefinitionRef":"qdrant","componentSpecs":[{"name":"qdrant","replicas":2,"resources":{"limits":{"cpu":"100m","memory":"0.5Gi"},"requests":{"cpu":"100m","memory":"0.5Gi"}},"serviceAccountName":"kb-qdrant-cluster","serviceVersion":"1.5.0","volumeClaimTemplates":[{"name":"data","spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"20Gi"}},"storageClassName":null}}]}],"terminationPolicy":"WipeOut","topology":"cluster"}}
creationTimestamp: "2025-04-28T07:23:40Z"
finalizers:
- cluster.kubeblocks.io/finalizer
generation: 3
labels:
clusterdefinition.kubeblocks.io/name: qdrant
clusterversion.kubeblocks.io/name: ""
name: qdrant-cluster
namespace: default
resourceVersion: "59393"
uid: 9ef5913e-c942-423f-8d65-c24f96035f2a
spec:
clusterDef: qdrant
componentSpecs:
- componentDef: qdrant-1.0.0-alpha.0
name: qdrant
replicas: 2
resources:
limits:
cpu: 100m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
serviceVersion: 1.5.0
volumeClaimTemplates:
- name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
terminationPolicy: WipeOut
topology: cluster
status:
components:
qdrant:
phase: Updating
conditions:
- lastTransitionTime: "2025-04-28T08:07:35Z"
message: 'The operator has started the provisioning of Cluster: qdrant-cluster'
observedGeneration: 3
reason: PreCheckSucceed
status: "True"
type: ProvisioningStarted
- lastTransitionTime: "2025-04-28T07:23:42Z"
message: Successfully applied for resources
observedGeneration: 3
reason: ApplyResourcesSucceed
status: "True"
type: ApplyResources
- lastTransitionTime: "2025-04-28T07:24:23Z"
message: all pods of components are ready, waiting for the probe detection successful
reason: AllReplicasReady
status: "True"
type: ReplicasReady
- lastTransitionTime: "2025-04-28T07:24:23Z"
message: 'Cluster: qdrant-cluster is ready, current phase is Running'
reason: ClusterReady
status: "True"
type: Ready
observedGeneration: 3
phase: Updating
kubectl get cluster qdrant-cluster
NAME CLUSTER-DEFINITION TERMINATION-POLICY STATUS AGE
qdrant-cluster qdrant WipeOut Updating 74m
➜ ~
➜ ~ kubectl get pod -l app.kubernetes.io/instance=qdrant-cluster
NAME READY STATUS RESTARTS AGE
qdrant-cluster-qdrant-0 2/2 Running 0 74m
qdrant-cluster-qdrant-1 1/2 Running 4 (52s ago) 15m
➜ ~
➜ ~ kubectl logs qdrant-cluster-qdrant-1 --previous
Defaulted container "qdrant" out of: qdrant, kbagent, qdrant-tools (init), init-kbagent (init), kbagent-worker (init)
BOOTSTRAP_HOSTNAME: ..svc
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: ..svc
INFO: wait for bootstrap node starting...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: ..svc
INFO: wait for bootstrap node starting...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: ..svc
INFO: wait for bootstrap node starting...
% Total % Received % Xferd Average Speed Time Time Time Current