milvus
milvus copied to clipboard
[Bug]: [Nightly] Drop a partition of a loaded collection will fail for timeout sometimes
Is there an existing issue for this?
- [X] I have searched the existing issues
Environment
- Milvus version:897ed62
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka): rocksmq
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0.dev12
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:
Current Behavior
Drop a partition of a loaded collection will fail for timeout sometimes.
[2023-04-22T21:44:43.353Z] [2023-04-22 17:58:52 - ERROR - pymilvus.decorators]: RPC error: [drop_partition], <MilvusException: (code=1, message=rpc deadline exceeded: Retry timeout: 20s)>, <Time:{'RPC start': '2023-04-22 17:58:32.126988', 'RPC error': '2023-04-22 17:58:52.127614'}> (decorators.py:108)
[2023-04-22T21:44:43.353Z] [2023-04-22 17:58:52 - ERROR - ci_test]: Traceback (most recent call last):
[2023-04-22T21:44:43.353Z] File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 50, in handler
[2023-04-22T21:44:43.353Z] return func(self, *args, **kwargs)
[2023-04-22T21:44:43.353Z] File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 299, in drop_partition
[2023-04-22T21:44:43.353Z] response = rf.result()
[2023-04-22T21:44:43.353Z] File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 797, in result
[2023-04-22T21:44:43.353Z] raise self
[2023-04-22T21:44:43.353Z] grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
[2023-04-22T21:44:43.353Z] status = StatusCode.DEADLINE_EXCEEDED
[2023-04-22T21:44:43.353Z] details = "Deadline Exceeded"
[2023-04-22T21:44:43.353Z] debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2023-04-22T17:58:52.127249378+00:00", grpc_status:4}"
[2023-04-22T21:44:43.353Z] >
[2023-04-22T21:44:43.353Z]
[2023-04-22T21:44:43.353Z] The above exception was the direct cause of the following exception:
[2023-04-22T21:44:43.353Z]
[2023-04-22T21:44:43.353Z] Traceback (most recent call last):
[2023-04-22T21:44:43.353Z] File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 26, in inner_wrapper
[2023-04-22T21:44:43.353Z] res = func(*args, **_kwargs)
[2023-04-22T21:44:43.353Z] File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 57, in api_request
[2023-04-22T21:44:43.353Z] return func(*arg, **kwargs)
[2023-04-22T21:44:43.353Z] File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/partition.py", line 190, in drop
[2023-04-22T21:44:43.353Z] return conn.drop_partition(self._collection.name, self._name, timeout=timeout, **kwargs)
[2023-04-22T21:44:43.353Z] File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 109, in handler
[2023-04-22T21:44:43.353Z] raise e
[2023-04-22T21:44:43.353Z] File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 105, in handler
[2023-04-22T21:44:43.353Z] return func(*args, **kwargs)
[2023-04-22T21:44:43.354Z] File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2023-04-22T21:44:43.354Z] ret = func(self, *args, **kwargs)
[2023-04-22T21:44:43.354Z] File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 64, in handler
[2023-04-22T21:44:43.354Z] raise MilvusException(message=f"rpc deadline exceeded: {timeout_msg}") from e
[2023-04-22T21:44:43.354Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=rpc deadline exceeded: Retry timeout: 20s)>
[2023-04-22T21:44:43.354Z] (api_request.py:39)
[2023-04-22T21:44:43.354Z] [2023-04-22 17:58:52 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=rpc deadline exceeded: Retry timeout: 20s)> (api_request.py:40)
Expected Behavior
pass
Steps To Reproduce
1. create a collection and 2 partitions
2. create index and load collection
3. drop a partition ❌
Milvus Log
- link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/master/350/pipeline/123/
- log: artifacts-milvus-standalone-nightly-350-pymilvus-e2e-logs.tar.gz
- collection name: collection_fkMU43kZ
- failed time: [2023-04-22T17:59:02.404Z] [gw4] [ 6%] FAILED testcases/test_collection.py::TestLoadCollection::test_load_another_partition_after_drop_one_partition
Anything else?
No response