[Bug]: Load collection timeout without any interruption

[zhuwenxing]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version:2.2.0-20230404-4e347518
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):kafka    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

2023-04-06T23:37:55.840Z] [2023-04-06 23:35:08 - DEBUG - ci_test]: (api_request)  : [Collection.create_index] args: ['varchar', {}], kwargs: {'timeout': 1200, 'index_name': 'test_piO6wtP7'} (api_request.py:56)

[2023-04-06T23:37:55.840Z] [2023-04-06 23:35:44 - DEBUG - ci_test]: (api_response) : Status(code=0, message=)  (api_request.py:31)

[2023-04-06T23:37:55.840Z] [2023-04-06 23:35:44 - INFO - ci_test]: [test][2023-04-06T23:35:08Z] [35.73951267s] e2e__vTgWj9Jb create_index -> Status(code=0, message=) (wrapper.py:30)

[2023-04-06T23:37:55.840Z] [2023-04-06 23:35:44 - INFO - ci_test]: assert index: 73.05894947052002 (test_e2e.py:53)

[2023-04-06T23:37:55.840Z] [2023-04-06 23:35:44 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 120], kwargs: {} (api_request.py:56)

[2023-04-06T23:37:55.840Z] [2023-04-06 23:37:45 - ERROR - pymilvus.decorators]: RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=wait for loading collection timeout, collection: e2e__vTgWj9Jb)>, <Time:{'RPC start': '2023-04-06 23:35:44.695270', 'RPC error': '2023-04-06 23:37:45.173804'}> (decorators.py:108)

[2023-04-06T23:37:55.840Z] [2023-04-06 23:37:45 - WARNING - pymilvus.decorators]: Retry timeout: 120s (decorators.py:79)

[2023-04-06T23:37:55.840Z] [2023-04-06 23:37:45 - ERROR - pymilvus.decorators]: RPC error: [load_collection], <MilvusException: (code=1, message=Retry timeout: 120s, message=wait for loading collection timeout, collection: e2e__vTgWj9Jb)>, <Time:{'RPC start': '2023-04-06 23:35:44.280102', 'RPC error': '2023-04-06 23:37:45.174241'}> (decorators.py:108)

[2023-04-06T23:37:55.840Z] [2023-04-06 23:37:45 - ERROR - ci_test]: Traceback (most recent call last):

[2023-04-06T23:37:55.840Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 50, in handler

[2023-04-06T23:37:55.840Z]     return func(self, *args, **kwargs)

[2023-04-06T23:37:55.840Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 710, in load_collection

[2023-04-06T23:37:55.840Z]     self.wait_for_loading_collection(collection_name, timeout)

[2023-04-06T23:37:55.840Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 109, in handler

[2023-04-06T23:37:55.840Z]     raise e

[2023-04-06T23:37:55.840Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 105, in handler

[2023-04-06T23:37:55.840Z]     return func(*args, **kwargs)

[2023-04-06T23:37:55.840Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 136, in handler

[2023-04-06T23:37:55.840Z]     ret = func(self, *args, **kwargs)

[2023-04-06T23:37:55.840Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 85, in handler

[2023-04-06T23:37:55.840Z]     raise e

[2023-04-06T23:37:55.840Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 50, in handler

[2023-04-06T23:37:55.840Z]     return func(self, *args, **kwargs)

[2023-04-06T23:37:55.840Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 732, in wait_for_loading_collection

[2023-04-06T23:37:55.840Z]     raise MilvusException(message=f"wait for loading collection timeout, collection: {collection_name}")

[2023-04-06T23:37:55.840Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=wait for loading collection timeout, collection: e2e__vTgWj9Jb)>

[2023-04-06T23:37:55.840Z] 

[2023-04-06T23:37:55.840Z] The above exception was the direct cause of the following exception:

[2023-04-06T23:37:55.840Z] 

[2023-04-06T23:37:55.840Z] Traceback (most recent call last):

[2023-04-06T23:37:55.840Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 26, in inner_wrapper

[2023-04-06T23:37:55.840Z]     res = func(*args, **_kwargs)

[2023-04-06T23:37:55.840Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 57, in api_request

[2023-04-06T23:37:55.840Z]     return func(*arg, **kwargs)

[2023-04-06T23:37:55.840Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 366, in load

[2023-04-06T23:37:55.840Z]     conn.load_collection(self._name, replica_number=replica_number, timeout=timeout, **kwargs)

[2023-04-06T23:37:55.840Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 109, in handler

[2023-04-06T23:37:55.840Z]     raise e

[2023-04-06T23:37:55.840Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 105, in handler

[2023-04-06T23:37:55.840Z]     return func(*args, **kwargs)

[2023-04-06T23:37:55.840Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 136, in handler

[2023-04-06T23:37:55.840Z]     ret = func(self, *args, **kwargs)

[2023-04-06T23:37:55.840Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 80, in handler

[2023-04-06T23:37:55.840Z]     raise MilvusException(e.code, f"{timeout_msg}, message={e.message}") from e

[2023-04-06T23:37:55.840Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=Retry timeout: 120s, message=wait for loading collection timeout, collection: e2e__vTgWj9Jb)>

[2023-04-06T23:37:55.840Z]  (api_request.py:39)

[2023-04-06T23:37:55.840Z] [2023-04-06 23:37:45 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=Retry timeout: 120s, message=wait for loading collection timeout, collection: e2e__vTgWj9Jb)> (api_request.py:40)

[2023-04-06T23:37:55.840Z] ------------- generated html file: file:///tmp/ci_logs/report.html -------------

[2023-04-06T23:37:55.840Z] =========================== short test summary info ============================

[2023-04-06T23:37:55.840Z] FAILED ../testcases/test_e2e.py::TestE2e::test_milvus_default - AssertionError

[2023-04-06T23:37:55.840Z] ======================== 1 failed in 221.88s (0:03:41) =========================

Expected Behavior

all test cases pass

Steps To Reproduce

No response

Milvus Log

image tag: 2.2.0-20230404-4e347518 failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-kafka-for-release-cron/detail/chaos-test-kafka-for-release-cron/3285/pipeline log: artifacts-querycoord-pod-failure-3285-server-logs.tar.gz

artifacts-querycoord-pod-failure-3285-pytest-logs.tar.gz

Anything else?

other same failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-kafka-for-release-cron/detail/chaos-test-kafka-for-release-cron/3287/pipeline log: artifacts-rootcoord-pod-failure-3287-server-logs.tar.gz artifacts-rootcoord-pod-failure-3287-pytest-logs.tar.gz

[yanliang567]

/assign @jiaoew1991 /unassign

[jiaoew1991]

/assign @sunby /unassign

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.