[BUG] qdrant upgrade version ERROR /qdrant.Raft/Send unexpectedly failed with Internal error Can't send Raft message over channel
Describe the bug A clear and concise description of what the bug is.
kbcli version
Kubernetes: v1.30.4-vke.4
KubeBlocks: 1.0.1-beta.19
kbcli: 1.0.1-beta.2
To Reproduce Steps to reproduce the behavior:
- create cluster
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
name: qdrant-sjsyky
namespace: default
spec:
clusterDef: qdrant
topology: cluster
terminationPolicy: WipeOut
componentSpecs:
- name: qdrant
serviceVersion: 1.5.0
replicas: 2
disableExporter: true
resources:
requests:
cpu: 100m
memory: 0.5Gi
limits:
cpu: 100m
memory: 0.5Gi
volumeClaimTemplates:
- name: data
spec:
storageClassName:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
- upgrade version
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
generateName: qdrant-sjsyky-upgrade-cmpv-
namespace: default
spec:
clusterName: qdrant-sjsyky
upgrade:
components:
- componentName: qdrant
serviceVersion: 1.10.0
type: Upgrade
- See error
kubectl get cluster qdrant-sjsyky
NAME CLUSTER-DEFINITION TERMINATION-POLICY STATUS AGE
qdrant-sjsyky qdrant WipeOut Updating 19m
➜ ~
➜ ~ kubectl get cmp -l app.kubernetes.io/instance=qdrant-sjsyky
NAME DEFINITION SERVICE-VERSION STATUS AGE
qdrant-sjsyky-qdrant qdrant-1.0.1-beta.0 1.10.0 Updating 19m
➜ ~
➜ ~ kubectl get pod -l app.kubernetes.io/instance=qdrant-sjsyky
NAME READY STATUS RESTARTS AGE
qdrant-sjsyky-qdrant-0 1/2 Running 0 11m
qdrant-sjsyky-qdrant-1 2/2 Running 1 (9m4s ago) 11m
➜ ~
➜ ~ kubectl get ops -l app.kubernetes.io/instance=qdrant-sjsyky
NAME TYPE CLUSTER STATUS PROGRESS AGE
qdrant-sjsyky-upgrade-cmpv-vlxgv Upgrade qdrant-sjsyky Running 1/2 9m44s
describe ops
kubectl describe ops qdrant-sjsyky-upgrade-cmpv-vlxgv
Name: qdrant-sjsyky-upgrade-cmpv-vlxgv
Namespace: default
Labels: app.kubernetes.io/instance=qdrant-sjsyky
operations.kubeblocks.io/ops-type=Upgrade
Annotations: <none>
API Version: operations.kubeblocks.io/v1alpha1
Kind: OpsRequest
Metadata:
Creation Timestamp: 2025-09-08T09:11:18Z
Finalizers:
opsrequest.kubeblocks.io/finalizer
Generate Name: qdrant-sjsyky-upgrade-cmpv-
Generation: 1
Owner References:
API Version: apps.kubeblocks.io/v1
Kind: Cluster
Name: qdrant-sjsyky
UID: e503ae7d-7af2-46c2-af76-1c10bdb0e9e1
Resource Version: 91154080
UID: f06272d9-a8b3-4df5-ae36-1985f0682bab
Spec:
Cluster Name: qdrant-sjsyky
Enqueue On Force: false
Pre Condition Deadline Seconds: 0
Type: Upgrade
Upgrade:
Components:
Component Name: qdrant
Service Version: 1.10.0
Status:
Cluster Generation: 8
Components:
Qdrant:
Phase: Updating
Progress Details:
Object Key: Pod/qdrant-sjsyky-qdrant-0
Status: Pending
End Time: 2025-09-08T09:12:20Z
Message: Successfully upgrade: Pod/qdrant-sjsyky-qdrant-1 in Component: qdrant
Object Key: Pod/qdrant-sjsyky-qdrant-1
Start Time: 2025-09-08T09:12:10Z
Status: Succeed
Conditions:
Last Transition Time: 2025-09-08T09:11:18Z
Message: wait for the controller to process the OpsRequest: qdrant-sjsyky-upgrade-cmpv-vlxgv in Cluster: qdrant-sjsyky
Reason: WaitForProgressing
Status: True
Type: WaitForProgressing
Last Transition Time: 2025-09-08T09:11:18Z
Message: OpsRequest: qdrant-sjsyky-upgrade-cmpv-vlxgv is validated
Reason: ValidateOpsRequestPassed
Status: True
Type: Validated
Last Transition Time: 2025-09-08T09:11:18Z
Message: Start to upgrade the version in Cluster: qdrant-sjsyky
Reason: VersionUpgradeStarted
Status: True
Type: VersionUpgrading
Last Configuration:
Components:
Qdrant:
Component Definition Name: qdrant-1.0.1-beta.0
Service Version: 1.5.0
Phase: Running
Progress: 1/2
Start Timestamp: 2025-09-08T09:11:18Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForProgressing 9m56s ops-request-controller wait for the controller to process the OpsRequest: qdrant-sjsyky-upgrade-cmpv-vlxgv in Cluster: qdrant-sjsyky
Normal ValidateOpsRequestPassed 9m56s ops-request-controller OpsRequest: qdrant-sjsyky-upgrade-cmpv-vlxgv is validated
Normal VersionUpgradeStarted 9m56s ops-request-controller Start to upgrade the version in Cluster: qdrant-sjsyky
Normal Processing 9m4s ops-request-controller Start to upgrade: Pod/qdrant-sjsyky-qdrant-1 in Component: qdrant
Normal Succeed 8m54s ops-request-controller Successfully upgrade: Pod/qdrant-sjsyky-qdrant-1 in Component: qdrant
logs pod
kubectl logs qdrant-sjsyky-qdrant-0
Defaulted container "qdrant" out of: qdrant, kbagent, qdrant-tools (init), init-kbagent (init), kbagent-worker (init)
_ _
__ _ __| |_ __ __ _ _ __ | |_
/ _` |/ _` | '__/ _` | '_ \| __|
| (_| | (_| | | | (_| | | | | |_
\__, |\__,_|_| \__,_|_| |_|\__|
|_|
Access web UI at http://localhost:6333/dashboard
2025-09-08T09:09:21.124280Z INFO storage::content_manager::consensus::persistent: Loading raft state from ./storage/raft_state.json
2025-09-08T09:09:21.125056Z INFO storage::content_manager::toc: Loading collection: collection_qyufw
2025-09-08T09:09:21.326020Z INFO storage::content_manager::toc: Loading collection: executions_loop_collection
2025-09-08T09:09:21.534478Z INFO qdrant: Telemetry reporting disabled
2025-09-08T09:09:21.534653Z INFO qdrant::tonic: TLS disabled for internal gRPC API
2025-09-08T09:09:21.618005Z INFO qdrant::actix: TLS disabled for REST API
2025-09-08T09:09:21.618051Z INFO qdrant::actix: Qdrant HTTP listening on 6333
2025-09-08T09:09:21.618055Z INFO actix_server::builder: Starting 1 workers
2025-09-08T09:09:21.618064Z INFO actix_server::server: Actix runtime found; starting in Actix runtime
2025-09-08T09:09:21.618471Z INFO qdrant::tonic: Qdrant gRPC listening on 6334
2025-09-08T09:09:21.618479Z INFO qdrant::tonic: TLS disabled for gRPC API
2025-09-08T09:09:24.873125Z WARN storage::content_manager::consensus_manager: Failed to send message to http://qdrant-sjsyky-qdrant-1.qdrant-sjsyky-qdrant-headless.default.svc.cluster.local:6335/ with error: Error in closure supplied to transport channel pool: status: Unavailable, message: "Failed to connect to http://qdrant-sjsyky-qdrant-1.qdrant-sjsyky-qdrant-headless.default.svc.cluster.local:6335/, error: transport error", details: [], metadata: MetadataMap { headers: {} }
2025-09-08T09:09:27.814338Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 981 "-" "curl/8.1.1" 0.000095
2025-09-08T09:09:32.349407Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 981 "-" "curl/8.1.1" 0.000067
2025-09-08T09:09:44.125822Z INFO actix_web::middleware::logger: 192.168.0.245 "GET /cluster HTTP/1.1" 200 981 "-" "curl/8.1.1" 0.000068
2025-09-08T09:09:47.349284Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 980 "-" "curl/8.1.1" 0.000059
2025-09-08T09:10:02.348219Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 595 "-" "curl/8.1.1" 0.000057
2025-09-08T09:10:03.718012Z INFO actix_web::middleware::logger: 192.168.0.246 "POST /collections/executions_loop_collection/points/search HTTP/1.1" 200 111 "-" "curl/8.1.1" 0.003021
2025-09-08T09:10:14.277599Z INFO actix_web::middleware::logger: 192.168.0.241 "GET /cluster HTTP/1.1" 200 594 "-" "curl/8.1.1" 0.000068
2025-09-08T09:10:14.380627Z WARN storage::content_manager::consensus_manager: Failed to send message to http://qdrant-sjsyky-qdrant-2.qdrant-sjsyky-qdrant-headless.default.svc.cluster.local:6335/ with error: Error in closure supplied to transport channel pool: status: Unavailable, message: "Failed to connect to http://qdrant-sjsyky-qdrant-2.qdrant-sjsyky-qdrant-headless.default.svc.cluster.local:6335/, error: transport error", details: [], metadata: MetadataMap { headers: {} }
2025-09-08T09:10:17.346454Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 716 "-" "curl/8.1.1" 0.000052
2025-09-08T09:10:32.350367Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 716 "-" "curl/8.1.1" 0.000054
2025-09-08T09:10:38.207781Z INFO actix_web::middleware::logger: 192.168.0.246 "POST /collections/executions_loop_collection/points/search HTTP/1.1" 200 111 "-" "curl/8.1.1" 0.000597
2025-09-08T09:10:39.178109Z WARN qdrant::consensus: Failed to step message: StepPeerNotFound
2025-09-08T09:10:47.350335Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 594 "-" "curl/8.1.1" 0.000060
2025-09-08T09:11:02.347525Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 595 "-" "curl/8.1.1" 0.000065
2025-09-08T09:11:17.350436Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 595 "-" "curl/8.1.1" 0.000057
2025-09-08T09:11:18.017959Z INFO actix_web::middleware::logger: 192.168.0.246 "POST /collections/executions_loop_collection/points/search HTTP/1.1" 200 111 "-" "curl/8.1.1" 0.041250
2025-09-08T09:11:32.346207Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 595 "-" "curl/8.1.1" 0.000054
2025-09-08T09:11:47.352327Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 595 "-" "curl/8.1.1" 0.000057
2025-09-08T09:11:48.749368Z WARN storage::content_manager::consensus_manager: Failed to send message to http://qdrant-sjsyky-qdrant-1.qdrant-sjsyky-qdrant-headless.default.svc.cluster.local:6335/ with error: Error in closure supplied to transport channel pool: status: Unavailable, message: "error trying to connect: tcp connect error: Connection refused (os error 111)", details: [], metadata: MetadataMap { headers: {} }
2025-09-08T09:11:57.122457Z INFO actix_web::middleware::logger: 192.168.0.245 "GET /cluster HTTP/1.1" 200 994 "-" "curl/8.1.1" 0.000065
2025-09-08T09:11:57.523968Z ERROR qdrant::tonic::logging: gRPC /qdrant.QdrantInternal/GetConsensusCommit unexpectedly failed with Operation is not implemented or not supported "" 0.000020
2025-09-08T09:11:57.754296Z ERROR qdrant::consensus: Consensus stopped with error: Failed to handle committed entries: Failed to apply collection meta operation entry
2025-09-08T09:11:57.761941Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000019
2025-09-08T09:12:00.825708Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000045
2025-09-08T09:12:02.348228Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 649 "-" "curl/8.1.1" 0.000050
2025-09-08T09:12:02.826338Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000043
2025-09-08T09:12:06.027167Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000045
2025-09-08T09:12:08.326097Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000043
2025-09-08T09:12:10.926448Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000041
2025-09-08T09:12:13.625816Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000044
2025-09-08T09:12:17.226407Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000039
2025-09-08T09:12:17.350191Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 648 "-" "curl/8.1.1" 0.000069
2025-09-08T09:12:17.520546Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 648 "-" "curl/8.1.1" 0.000060
2025-09-08T09:12:19.826097Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000042
2025-09-08T09:12:22.227343Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000038
2025-09-08T09:12:25.325579Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000043
2025-09-08T09:12:28.225589Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000043
2025-09-08T09:12:32.026787Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000043
2025-09-08T09:12:32.351441Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 649 "-" "curl/8.1.1" 0.000060
2025-09-08T09:12:35.125894Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000056
2025-09-08T09:12:38.426364Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000047
2025-09-08T09:12:41.126663Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000042
2025-09-08T09:12:44.725600Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000043
2025-09-08T09:12:47.225903Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000054
2025-09-08T09:12:47.346215Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 649 "-" "curl/8.1.1" 0.000056
2025-09-08T09:12:49.326110Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000043
2025-09-08T09:12:52.626155Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000047
2025-09-08T09:12:56.326324Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000047
2025-09-08T09:13:00.026948Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000039
2025-09-08T09:13:02.345202Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 649 "-" "curl/8.1.1" 0.000051
2025-09-08T09:13:03.326400Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000044
2025-09-08T09:13:05.825537Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000040
2025-09-08T09:13:08.027163Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000049
2025-09-08T09:13:11.526173Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000049
2025-09-08T09:13:14.126468Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000046
2025-09-08T09:13:16.426975Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000049
2025-09-08T09:13:17.348225Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 649 "-" "curl/8.1.1" 0.000055
2025-09-08T09:13:19.925922Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000043
2025-09-08T09:13:23.626219Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000042
2025-09-08T09:13:26.526848Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000039
2025-09-08T09:13:26.875212Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 649 "-" "curl/8.1.1" 0.000059
2025-09-08T09:13:29.226303Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000045
2025-09-08T09:13:32.349458Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 648 "-" "curl/8.1.1" 0.000050
2025-09-08T09:13:32.525961Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000046
2025-09-08T09:13:36.426120Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000046
2025-09-08T09:13:38.626435Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000041
2025-09-08T09:13:41.625644Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000041
2025-09-08T09:13:45.426432Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000038
2025-09-08T09:13:47.349376Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 648 "-" "curl/8.1.1" 0.000056
2025-09-08T09:13:49.225682Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000043
2025-09-08T09:13:52.825943Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000046
2025-09-08T09:13:55.619460Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000044
2025-09-08T09:13:58.526307Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000043
2025-09-08T09:14:00.625978Z ERROR qdrant::tonic::logging: gRPC /qdrant.Raft/Send unexpectedly failed with Internal error "Can't send Raft message over channel" 0.000043
2025-09-08T09:14:02.345260Z INFO actix_web::middleware::logger: 127.0.0.1 "GET /cluster HTTP/1.1" 200 649 "-" "curl/8.1.1" 0.000054
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
Additional context Add any other context about the problem here.
How do I avoid issues when updating to the latest version?
We only guarantee compatibility if you update between consecutive versions. You would need to upgrade versions one at a time: 1.1 -> 1.2, then 1.2 -> 1.3, then 1.3 -> 1.4.
https://qdrant.tech/documentation/faq/qdrant-fundamentals/#how-do-i-avoid-issues-when-updating-to-the-latest-version
This issue has been marked as stale because it has been open for 30 days with no activity