[Bug]: [Nightly]Reload a released partition will cause rpc deadline exceeded

[NicoYuan1986]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version: 72485c9
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): rocksmq
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0.dev12
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Reload a released partition will cause rpc deadline exceeded.

[2023-04-24T21:47:00.542Z] [2023-04-24 18:00:10 - ERROR - pymilvus.decorators]: RPC error: [query], <MilvusException: (code=1, message=checkIfLoaded failed when query, collection:collection_kEskKXzB, partitions:[partition1], err = GetCollectionInfo failed, collection = collection_kEskKXzB, err = collection 441020289419658556 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-04-24 18:00:10.813138', 'RPC error': '2023-04-24 18:00:10.815017'}> (decorators.py:108)
[2023-04-24T21:47:00.542Z] [2023-04-24 18:00:10 - ERROR - ci_test]: Traceback (most recent call last):
[2023-04-24T21:47:00.542Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 26, in inner_wrapper
[2023-04-24T21:47:00.542Z]     res = func(*args, **_kwargs)
[2023-04-24T21:47:00.542Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 57, in api_request
[2023-04-24T21:47:00.542Z]     return func(*arg, **kwargs)
[2023-04-24T21:47:00.542Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 744, in query
[2023-04-24T21:47:00.542Z]     res = conn.query(self._name, expr, output_fields, partition_names,
[2023-04-24T21:47:00.542Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 109, in handler
[2023-04-24T21:47:00.542Z]     raise e
[2023-04-24T21:47:00.542Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 105, in handler
[2023-04-24T21:47:00.542Z]     return func(*args, **kwargs)
[2023-04-24T21:47:00.542Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2023-04-24T21:47:00.542Z]     ret = func(self, *args, **kwargs)
[2023-04-24T21:47:00.542Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 85, in handler
[2023-04-24T21:47:00.542Z]     raise e
[2023-04-24T21:47:00.542Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 50, in handler
[2023-04-24T21:47:00.542Z]     return func(self, *args, **kwargs)
[2023-04-24T21:47:00.542Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 949, in query
[2023-04-24T21:47:00.542Z]     raise MilvusException(response.status.error_code, response.status.reason)
[2023-04-24T21:47:00.542Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=checkIfLoaded failed when query, collection:collection_kEskKXzB, partitions:[partition1], err = GetCollectionInfo failed, collection = collection_kEskKXzB, err = collection 441020289419658556 has not been loaded to memory or load failed)>
[2023-04-24T21:47:00.542Z]  (api_request.py:39)
[2023-04-24T21:47:00.542Z] [2023-04-24 18:00:10 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=checkIfLoaded failed when query, collection:collection_kEskKXzB, partitions:[partition1], err = GetCollectionInfo failed, collection = collection_kEskKXzB, err = collection 441020289419658556 has not been loaded to memory or load failed)> (api_request.py:40)
[2023-04-24T21:47:00.542Z] [2023-04-24 18:00:10 - DEBUG - ci_test]: (api_request)  : [Partition.load] args: [1, 20], kwargs: {} (api_request.py:56)
[2023-04-24T21:47:00.542Z] [2023-04-24 18:00:40 - ERROR - pymilvus.decorators]: RPC error: [load_partitions], <MilvusException: (code=1, message=rpc deadline exceeded: Retry timeout: 20s)>, <Time:{'RPC start': '2023-04-24 18:00:20.252934', 'RPC error': '2023-04-24 18:00:40.253993'}> (decorators.py:108)
[2023-04-24T21:47:00.542Z] [2023-04-24 18:00:40 - ERROR - ci_test]: Traceback (most recent call last):
[2023-04-24T21:47:00.542Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 50, in handler
[2023-04-24T21:47:00.542Z]     return func(self, *args, **kwargs)
[2023-04-24T21:47:00.542Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 769, in load_partitions
[2023-04-24T21:47:00.542Z]     response = future.result()
[2023-04-24T21:47:00.542Z]   File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 797, in result
[2023-04-24T21:47:00.542Z]     raise self
[2023-04-24T21:47:00.542Z] grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
[2023-04-24T21:47:00.542Z] 	status = StatusCode.DEADLINE_EXCEEDED
[2023-04-24T21:47:00.542Z] 	details = "Deadline Exceeded"
[2023-04-24T21:47:00.542Z] 	debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2023-04-24T18:00:40.253532314+00:00", grpc_status:4}"
[2023-04-24T21:47:00.542Z] >
[2023-04-24T21:47:00.542Z] 
[2023-04-24T21:47:00.542Z] The above exception was the direct cause of the following exception:
[2023-04-24T21:47:00.542Z] 
[2023-04-24T21:47:00.542Z] Traceback (most recent call last):
[2023-04-24T21:47:00.542Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 26, in inner_wrapper
[2023-04-24T21:47:00.542Z]     res = func(*args, **_kwargs)
[2023-04-24T21:47:00.542Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 57, in api_request
[2023-04-24T21:47:00.542Z]     return func(*arg, **kwargs)
[2023-04-24T21:47:00.542Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/partition.py", line 222, in load
[2023-04-24T21:47:00.542Z]     return conn.load_partitions(self._collection.name, [self._name], replica_number, timeout=timeout, **kwargs)
[2023-04-24T21:47:00.542Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 109, in handler
[2023-04-24T21:47:00.542Z]     raise e
[2023-04-24T21:47:00.542Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 105, in handler
[2023-04-24T21:47:00.542Z]     return func(*args, **kwargs)
[2023-04-24T21:47:00.542Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2023-04-24T21:47:00.542Z]     ret = func(self, *args, **kwargs)
[2023-04-24T21:47:00.542Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 64, in handler
[2023-04-24T21:47:00.542Z]     raise MilvusException(message=f"rpc deadline exceeded: {timeout_msg}") from e
[2023-04-24T21:47:00.542Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=rpc deadline exceeded: Retry timeout: 20s)>
[2023-04-24T21:47:00.542Z]  (api_request.py:39)
[2023-04-24T21:47:00.542Z] [2023-04-24 18:00:40 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=rpc deadline exceeded: Retry timeout: 20s)> (api_request.py:40)

Expected Behavior

pass

Steps To Reproduce

1. load partition
2. drop the unloaded partition
3. release the loaded partition
4. query on the released partition 
5. reload the partition ❌

Milvus Log

1. link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/master/352/pipeline/123/
2. log: artifacts-milvus-standalone-nightly-352-pymilvus-e2e-logs.tar.gz
3. collection name: collection_kEskKXzB
4. failed time: [2023-04-24T18:00:41.147Z] [gw4] [ 7%] FAILED testcases/test_collection.py::TestLoadPartition::test_release_load_partition_after_load_partition_drop_another

Anything else?

No response