milvus icon indicating copy to clipboard operation
milvus copied to clipboard

[Bug]: [streaming] When upgrading image from 2.5 to 2.6, loading collection failed: service resource streamingNode insufficient

Open ThreadDao opened this issue 5 months ago • 2 comments

Is there an existing issue for this?

  • [x] I have searched the existing issues

Environment

- Milvus version: zong-roll-ddl-5
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka): pulsar   
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

deploy 2.5 milvus with pulsar mq

client test

  1. create a collection -> index -> insert 10m -> flush -> index again -> load
  2. concurrent reqyests: insert + delete + flush + query + search + scene_search_test

upgrading image from 2.5 to master

  • during upgrading, load collection failed:
[2025-06-17 12:59:18,237 - ERROR - fouram]: (api_response) : [Collection.load] <MilvusException: (code=65535, message=call query coordinator LoadCollection: when load 1 replica count: service resource insufficient[currentStreamingNode=0][expectedStreamingNode=1])>, [requestId: 4761416f1c5044a0b20a724ba8ce9d16] (api_request.py:58)
  • during upgrading, many search / query requests timeout:
[2025-06-17 12:59:46,744 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-06-17T12:59:46.743899965+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
>, [requestId: 2dfea80cc5f1469fa8c086c5f99c95e1] (api_request.py:58)

Expected Behavior

No response

Steps To Reproduce

https://argo-workflows.zilliz.cc/archived-workflows/qa/7dede924-deb5-45ba-b8fa-53ce1a5a2462?nodeId=zong-roll-ddl-5

Milvus Log

pods:

zong-roll-ddl-5-milvus-datanode-7c7d5669c8-gcl4p                  1/1     Running       0               20h     10.104.32.122   4am-node39   <none>           <none>
zong-roll-ddl-5-milvus-datanode-7c7d5669c8-l5wv2                  1/1     Running       0               20h     10.104.9.192    4am-node14   <none>           <none>
zong-roll-ddl-5-milvus-mixcoord-7985556b88-5284w                  1/1     Running       0               20h     10.104.24.158   4am-node29   <none>           <none>
zong-roll-ddl-5-milvus-proxy-8d454d558-w72cx                      1/1     Running       0               20h     10.104.9.195    4am-node14   <none>           <none>
zong-roll-ddl-5-milvus-querynode-1-5dc55f9757-gr26j               1/1     Running       0               20h     10.104.27.207   4am-node31   <none>           <none>
zong-roll-ddl-5-milvus-querynode-1-5dc55f9757-rwldl               1/1     Running       0               20h     10.104.26.77    4am-node32   <none>           <none>
zong-roll-ddl-5-milvus-streamingnode-f855848db-f679x              1/1     Running       0               20h     10.104.24.157   4am-node29   <none>           <none>

Anything else?

No response

ThreadDao avatar Jun 18 '25 09:06 ThreadDao

will be fixed by https://github.com/milvus-io/milvus/pull/42822

ThreadDao avatar Jun 18 '25 09:06 ThreadDao

/assign @ThreadDao Please verify it as the pr merged /unassign

yanliang567 avatar Jun 18 '25 11:06 yanliang567

fixed master-20250625-336e743b-amd64 https://argo-workflows.zilliz.cc/archived-workflows/qa/4ce2f6b8-8f42-4b70-847c-ee36f758c416?nodeId=zong-roll-ddl-8

ThreadDao avatar Jun 26 '25 08:06 ThreadDao