milvus icon indicating copy to clipboard operation
milvus copied to clipboard

[Bug]: [Nightly] Drop a partition of a loaded collection will fail for timeout sometimes

Open NicoYuan1986 opened this issue 1 year ago • 0 comments

Is there an existing issue for this?

  • [X] I have searched the existing issues

Environment

- Milvus version:897ed62
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):   rocksmq 
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0.dev12
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Drop a partition of a loaded collection will fail for timeout sometimes.

[2023-04-22T21:44:43.353Z] [2023-04-22 17:58:52 - ERROR - pymilvus.decorators]: RPC error: [drop_partition], <MilvusException: (code=1, message=rpc deadline exceeded: Retry timeout: 20s)>, <Time:{'RPC start': '2023-04-22 17:58:32.126988', 'RPC error': '2023-04-22 17:58:52.127614'}> (decorators.py:108)
[2023-04-22T21:44:43.353Z] [2023-04-22 17:58:52 - ERROR - ci_test]: Traceback (most recent call last):
[2023-04-22T21:44:43.353Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 50, in handler
[2023-04-22T21:44:43.353Z]     return func(self, *args, **kwargs)
[2023-04-22T21:44:43.353Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 299, in drop_partition
[2023-04-22T21:44:43.353Z]     response = rf.result()
[2023-04-22T21:44:43.353Z]   File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 797, in result
[2023-04-22T21:44:43.353Z]     raise self
[2023-04-22T21:44:43.353Z] grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
[2023-04-22T21:44:43.353Z] 	status = StatusCode.DEADLINE_EXCEEDED
[2023-04-22T21:44:43.353Z] 	details = "Deadline Exceeded"
[2023-04-22T21:44:43.353Z] 	debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2023-04-22T17:58:52.127249378+00:00", grpc_status:4}"
[2023-04-22T21:44:43.353Z] >
[2023-04-22T21:44:43.353Z] 
[2023-04-22T21:44:43.353Z] The above exception was the direct cause of the following exception:
[2023-04-22T21:44:43.353Z] 
[2023-04-22T21:44:43.353Z] Traceback (most recent call last):
[2023-04-22T21:44:43.353Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 26, in inner_wrapper
[2023-04-22T21:44:43.353Z]     res = func(*args, **_kwargs)
[2023-04-22T21:44:43.353Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 57, in api_request
[2023-04-22T21:44:43.353Z]     return func(*arg, **kwargs)
[2023-04-22T21:44:43.353Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/partition.py", line 190, in drop
[2023-04-22T21:44:43.353Z]     return conn.drop_partition(self._collection.name, self._name, timeout=timeout, **kwargs)
[2023-04-22T21:44:43.353Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 109, in handler
[2023-04-22T21:44:43.353Z]     raise e
[2023-04-22T21:44:43.353Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 105, in handler
[2023-04-22T21:44:43.353Z]     return func(*args, **kwargs)
[2023-04-22T21:44:43.354Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2023-04-22T21:44:43.354Z]     ret = func(self, *args, **kwargs)
[2023-04-22T21:44:43.354Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 64, in handler
[2023-04-22T21:44:43.354Z]     raise MilvusException(message=f"rpc deadline exceeded: {timeout_msg}") from e
[2023-04-22T21:44:43.354Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=rpc deadline exceeded: Retry timeout: 20s)>
[2023-04-22T21:44:43.354Z]  (api_request.py:39)
[2023-04-22T21:44:43.354Z] [2023-04-22 17:58:52 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=rpc deadline exceeded: Retry timeout: 20s)> (api_request.py:40)

Expected Behavior

pass

Steps To Reproduce

1. create a collection and 2 partitions
2. create index and load collection
3. drop a partition ❌

Milvus Log

  1. link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/master/350/pipeline/123/
  2. log: artifacts-milvus-standalone-nightly-350-pymilvus-e2e-logs.tar.gz
  3. collection name: collection_fkMU43kZ
  4. failed time: [2023-04-22T17:59:02.404Z] [gw4] [ 6%] FAILED testcases/test_collection.py::TestLoadCollection::test_load_another_partition_after_drop_one_partition

Anything else?

No response

NicoYuan1986 avatar Apr 23 '23 07:04 NicoYuan1986