milvus
milvus copied to clipboard
[Bug]: [streaming] When upgrading image from 2.5 to 2.6, loading collection failed: service resource streamingNode insufficient
Is there an existing issue for this?
- [x] I have searched the existing issues
Environment
- Milvus version: zong-roll-ddl-5
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka): pulsar
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:
Current Behavior
deploy 2.5 milvus with pulsar mq
client test
- create a collection -> index -> insert 10m -> flush -> index again -> load
- concurrent reqyests: insert + delete + flush + query + search + scene_search_test
upgrading image from 2.5 to master
- during upgrading, load collection failed:
[2025-06-17 12:59:18,237 - ERROR - fouram]: (api_response) : [Collection.load] <MilvusException: (code=65535, message=call query coordinator LoadCollection: when load 1 replica count: service resource insufficient[currentStreamingNode=0][expectedStreamingNode=1])>, [requestId: 4761416f1c5044a0b20a724ba8ce9d16] (api_request.py:58)
- during upgrading, many search / query requests timeout:
[2025-06-17 12:59:46,744 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2025-06-17T12:59:46.743899965+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
>, [requestId: 2dfea80cc5f1469fa8c086c5f99c95e1] (api_request.py:58)
Expected Behavior
No response
Steps To Reproduce
https://argo-workflows.zilliz.cc/archived-workflows/qa/7dede924-deb5-45ba-b8fa-53ce1a5a2462?nodeId=zong-roll-ddl-5
Milvus Log
pods:
zong-roll-ddl-5-milvus-datanode-7c7d5669c8-gcl4p 1/1 Running 0 20h 10.104.32.122 4am-node39 <none> <none>
zong-roll-ddl-5-milvus-datanode-7c7d5669c8-l5wv2 1/1 Running 0 20h 10.104.9.192 4am-node14 <none> <none>
zong-roll-ddl-5-milvus-mixcoord-7985556b88-5284w 1/1 Running 0 20h 10.104.24.158 4am-node29 <none> <none>
zong-roll-ddl-5-milvus-proxy-8d454d558-w72cx 1/1 Running 0 20h 10.104.9.195 4am-node14 <none> <none>
zong-roll-ddl-5-milvus-querynode-1-5dc55f9757-gr26j 1/1 Running 0 20h 10.104.27.207 4am-node31 <none> <none>
zong-roll-ddl-5-milvus-querynode-1-5dc55f9757-rwldl 1/1 Running 0 20h 10.104.26.77 4am-node32 <none> <none>
zong-roll-ddl-5-milvus-streamingnode-f855848db-f679x 1/1 Running 0 20h 10.104.24.157 4am-node29 <none> <none>
Anything else?
No response
will be fixed by https://github.com/milvus-io/milvus/pull/42822
/assign @ThreadDao Please verify it as the pr merged /unassign
fixed master-20250625-336e743b-amd64
https://argo-workflows.zilliz.cc/archived-workflows/qa/4ce2f6b8-8f42-4b70-847c-ee36f758c416?nodeId=zong-roll-ddl-8