[Bug]: [Nightly] Create index will occasionally fail for timeout

[NicoYuan1986]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version: 2.3-20240321-f8496dbc-amd64
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    pulsar
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Create index will occasionally fail for timeout

[2024-03-21T23:25:07.573Z] [2024-03-21 22:44:00 - DEBUG - ci_test]: (api_request)  : [Collection.create_index] args: ['binary_vector', {'index_type': 'BIN_IVF_FLAT', 'params': {'nlist': 128}, 'metric_type': 'JACCARD'}, 1200], kwargs: {'index_name': ''} (api_request.py:62)
[2024-03-21T23:25:07.574Z] [2024-03-21 23:04:01 - WARNING - pymilvus.decorators]: [93mRetry timeout: 1200s[0m (decorators.py:105)
[2024-03-21T23:25:07.574Z] [2024-03-21 23:04:01 - ERROR - pymilvus.decorators]: RPC error: [wait_for_creating_index], <MilvusException: (code=1, message=Retry timeout: 1200s, message=collection search_collection_7zrPDxcQ create index  timeout in 1200s)>, <Time:{'RPC start': '2024-03-21 22:44:00.568749', 'RPC error': '2024-03-21 23:04:01.071991'}> (decorators.py:139)
[2024-03-21T23:25:07.574Z] [2024-03-21 23:04:01 - WARNING - pymilvus.decorators]: [93mRetry timeout: 1200s[0m (decorators.py:105)
[2024-03-21T23:25:07.574Z] [2024-03-21 23:04:01 - ERROR - pymilvus.decorators]: RPC error: [create_index], <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection search_collection_7zrPDxcQ create index  timeout in 1200s)>, <Time:{'RPC start': '2024-03-21 22:44:00.424285', 'RPC error': '2024-03-21 23:04:01.072169'}> (decorators.py:139)
[2024-03-21T23:25:07.574Z] [2024-03-21 23:04:01 - ERROR - ci_test]: Traceback (most recent call last):
[2024-03-21T23:25:07.574Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-03-21T23:25:07.574Z]     res = func(*args, **_kwargs)
[2024-03-21T23:25:07.574Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-03-21T23:25:07.574Z]     return func(*arg, **kwargs)
[2024-03-21T23:25:07.574Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 1218, in create_index
[2024-03-21T23:25:07.574Z]     return conn.create_index(self._name, field_name, index_params, timeout=timeout, **kwargs)
[2024-03-21T23:25:07.574Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 140, in handler
[2024-03-21T23:25:07.574Z]     raise e from e
[2024-03-21T23:25:07.574Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2024-03-21T23:25:07.574Z]     return func(*args, **kwargs)
[2024-03-21T23:25:07.574Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 175, in handler
[2024-03-21T23:25:07.574Z]     return func(self, *args, **kwargs)
[2024-03-21T23:25:07.574Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 106, in handler
[2024-03-21T23:25:07.574Z]     raise MilvusException(
[2024-03-21T23:25:07.574Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection search_collection_7zrPDxcQ create index  timeout in 1200s)>
[2024-03-21T23:25:07.574Z]  (api_request.py:45)
[2024-03-21T23:25:07.574Z] [2024-03-21 23:04:01 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection search_collection_7zrPDxcQ create index  timeout in 1200s)> (api_request.py:46)[create_path] folder(/tmp/ci_logs/test) is not exist.

Expected Behavior

create index successfully

Steps To Reproduce

No response

Milvus Log

1. link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/2.3/140/pipeline/237/
2. log: artifacts-milvus-distributed-pulsar-nightly-140-pymilvus-e2e-logs.tar.gz
3. collection name: search_collection_7zrPDxcQ
4. failed time: [2024-03-21T23:04:02.195Z] [gw5] [ 80%] FAILED testcases/test_search.py::TestSearchString::test_search_mix_expr_with_binary[32-False-False]

Anything else?

No response

[yanliang567]

/assign @xiaocai2333 I have assigned a few issues about index failures, please help /unassign

[NicoYuan1986]

update:

1. link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/2.4/7/pipeline/243
2. log: artifacts-milvus-distributed-kafka-nightly-7-pymilvus-e2e-logs.tar.gz
3. collection name: index_ueNllJHz
4. failed time: [2024-03-24T19:45:47.578Z] [gw2] [ 32%] FAILED testcases/test_index.py::TestIndexDiskann::test_create_diskann_index_multithread
5. error message:

[2024-03-24T22:19:54.863Z] [2024-03-24 19:25:46 - DEBUG - ci_test]: (api_request)  : [Collection.create_index] args: ['float_vector', {'index_type': 'DISKANN', 'metric_type': 'COSINE', 'params': {}}, 1200], kwargs: {'index_name': ''} (api_request.py:62)
[2024-03-24T22:19:54.863Z] [2024-03-24 19:25:46 - DEBUG - ci_test]: (api_request)  : [Collection.create_index] args: ['float_vector', {'index_type': 'DISKANN', 'metric_type': 'COSINE', 'params': {}}, 1200], kwargs: {'index_name': ''} (api_request.py:62)
[2024-03-24T22:19:54.863Z] [2024-03-24 19:25:46 - DEBUG - ci_test]: (api_request)  : [Collection.create_index] args: ['float_vector', {'index_type': 'DISKANN', 'metric_type': 'COSINE', 'params': {}}, 1200], kwargs: {'index_name': ''} (api_request.py:62)
[2024-03-24T22:19:54.863Z] [2024-03-24 19:25:46 - DEBUG - ci_test]: (api_request)  : [Collection.create_index] args: ['float_vector', {'index_type': 'DISKANN', 'metric_type': 'COSINE', 'params': {}}, 1200], kwargs: {'index_name': ''} (api_request.py:62)
[2024-03-24T22:19:54.863Z] [2024-03-24 19:25:46 - DEBUG - ci_test]: (api_request)  : [Collection.create_index] args: ['float_vector', {'index_type': 'DISKANN', 'metric_type': 'COSINE', 'params': {}}, 1200], kwargs: {'index_name': ''} (api_request.py:62)
[2024-03-24T22:19:54.863Z] [2024-03-24 19:25:47 - DEBUG - ci_test]: (api_request)  : [Collection.create_index] args: ['float_vector', {'index_type': 'DISKANN', 'metric_type': 'COSINE', 'params': {}}, 1200], kwargs: {'index_name': ''} (api_request.py:62)
[2024-03-24T22:19:54.863Z] [2024-03-24 19:25:47 - DEBUG - ci_test]: (api_request)  : [Collection.create_index] args: ['float_vector', {'index_type': 'DISKANN', 'metric_type': 'COSINE', 'params': {}}, 1200], kwargs: {'index_name': ''} (api_request.py:62)
[2024-03-24T22:19:54.863Z] [2024-03-24 19:25:47 - DEBUG - ci_test]: (api_request)  : [Collection.create_index] args: ['float_vector', {'index_type': 'DISKANN', 'metric_type': 'COSINE', 'params': {}}, 1200], kwargs: {'index_name': ''} (api_request.py:62)
[2024-03-24T22:19:54.863Z] [2024-03-24 19:25:47 - DEBUG - ci_test]: (api_request)  : [Collection.create_index] args: ['float_vector', {'index_type': 'DISKANN', 'metric_type': 'COSINE', 'params': {}}, 1200], kwargs: {'index_name': ''} (api_request.py:62)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:25:47 - DEBUG - ci_test]: (api_request)  : [Collection.create_index] args: ['float_vector', {'index_type': 'DISKANN', 'metric_type': 'COSINE', 'params': {}}, 1200], kwargs: {'index_name': ''} (api_request.py:62)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - WARNING - pymilvus.decorators]: [93mRetry timeout: 1200s[0m (decorators.py:105)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - ERROR - pymilvus.decorators]: RPC error: [wait_for_creating_index], <MilvusException: (code=1, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>, <Time:{'RPC start': '2024-03-24 19:25:46.909937', 'RPC error': '2024-03-24 19:45:47.260053'}> (decorators.py:139)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - WARNING - pymilvus.decorators]: [93mRetry timeout: 1200s[0m (decorators.py:105)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - ERROR - pymilvus.decorators]: RPC error: [create_index], <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>, <Time:{'RPC start': '2024-03-24 19:25:46.890199', 'RPC error': '2024-03-24 19:45:47.260444'}> (decorators.py:139)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - WARNING - pymilvus.decorators]: [93mRetry timeout: 1200s[0m (decorators.py:105)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - ERROR - pymilvus.decorators]: RPC error: [wait_for_creating_index], <MilvusException: (code=1, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>, <Time:{'RPC start': '2024-03-24 19:25:46.504435', 'RPC error': '2024-03-24 19:45:47.261216'}> (decorators.py:139)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - WARNING - pymilvus.decorators]: [93mRetry timeout: 1200s[0m (decorators.py:105)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - ERROR - pymilvus.decorators]: RPC error: [create_index], <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>, <Time:{'RPC start': '2024-03-24 19:25:46.488806', 'RPC error': '2024-03-24 19:45:47.261543'}> (decorators.py:139)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - WARNING - pymilvus.decorators]: [93mRetry timeout: 1200s[0m (decorators.py:105)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - ERROR - pymilvus.decorators]: RPC error: [wait_for_creating_index], <MilvusException: (code=1, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>, <Time:{'RPC start': '2024-03-24 19:25:46.864816', 'RPC error': '2024-03-24 19:45:47.262012'}> (decorators.py:139)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - WARNING - pymilvus.decorators]: [93mRetry timeout: 1200s[0m (decorators.py:105)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - ERROR - pymilvus.decorators]: RPC error: [create_index], <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>, <Time:{'RPC start': '2024-03-24 19:25:46.689624', 'RPC error': '2024-03-24 19:45:47.262205'}> (decorators.py:139)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - WARNING - pymilvus.decorators]: [93mRetry timeout: 1200s[0m (decorators.py:105)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - ERROR - pymilvus.decorators]: RPC error: [wait_for_creating_index], <MilvusException: (code=1, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>, <Time:{'RPC start': '2024-03-24 19:25:46.312497', 'RPC error': '2024-03-24 19:45:47.263239'}> (decorators.py:139)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - WARNING - pymilvus.decorators]: [93mRetry timeout: 1200s[0m (decorators.py:105)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - ERROR - pymilvus.decorators]: RPC error: [create_index], <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>, <Time:{'RPC start': '2024-03-24 19:25:46.288108', 'RPC error': '2024-03-24 19:45:47.263465'}> (decorators.py:139)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - WARNING - pymilvus.decorators]: [93mRetry timeout: 1200s[0m (decorators.py:105)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - ERROR - pymilvus.decorators]: RPC error: [wait_for_creating_index], <MilvusException: (code=1, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>, <Time:{'RPC start': '2024-03-24 19:25:47.154954', 'RPC error': '2024-03-24 19:45:47.264892'}> (decorators.py:139)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - WARNING - pymilvus.decorators]: [93mRetry timeout: 1200s[0m (decorators.py:105)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - ERROR - pymilvus.decorators]: RPC error: [create_index], <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>, <Time:{'RPC start': '2024-03-24 19:25:47.091037', 'RPC error': '2024-03-24 19:45:47.265078'}> (decorators.py:139)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - WARNING - pymilvus.decorators]: [93mRetry timeout: 1200s[0m (decorators.py:105)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - ERROR - pymilvus.decorators]: RPC error: [wait_for_creating_index], <MilvusException: (code=1, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>, <Time:{'RPC start': '2024-03-24 19:25:46.135917', 'RPC error': '2024-03-24 19:45:47.266225'}> (decorators.py:139)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - WARNING - pymilvus.decorators]: [93mRetry timeout: 1200s[0m (decorators.py:105)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - ERROR - pymilvus.decorators]: RPC error: [create_index], <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>, <Time:{'RPC start': '2024-03-24 19:25:46.087416', 'RPC error': '2024-03-24 19:45:47.266518'}> (decorators.py:139)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - ERROR - ci_test]: Traceback (most recent call last):
[2024-03-24T22:19:54.864Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-03-24T22:19:54.864Z]     res = func(*args, **_kwargs)
[2024-03-24T22:19:54.864Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-03-24T22:19:54.864Z]     return func(*arg, **kwargs)
[2024-03-24T22:19:54.864Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 1356, in create_index
[2024-03-24T22:19:54.864Z]     return conn.create_index(self._name, field_name, index_params, timeout=timeout, **kwargs)
[2024-03-24T22:19:54.864Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 140, in handler
[2024-03-24T22:19:54.864Z]     raise e from e
[2024-03-24T22:19:54.864Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2024-03-24T22:19:54.864Z]     return func(*args, **kwargs)
[2024-03-24T22:19:54.864Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 175, in handler
[2024-03-24T22:19:54.864Z]     return func(self, *args, **kwargs)
[2024-03-24T22:19:54.864Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 106, in handler
[2024-03-24T22:19:54.864Z]     raise MilvusException(
[2024-03-24T22:19:54.864Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>
[2024-03-24T22:19:54.864Z]  (api_request.py:45)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - ERROR - ci_test]: Traceback (most recent call last):
[2024-03-24T22:19:54.864Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-03-24T22:19:54.864Z]     res = func(*args, **_kwargs)
[2024-03-24T22:19:54.864Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-03-24T22:19:54.864Z]     return func(*arg, **kwargs)
[2024-03-24T22:19:54.864Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 1356, in create_index
[2024-03-24T22:19:54.864Z]     return conn.create_index(self._name, field_name, index_params, timeout=timeout, **kwargs)
[2024-03-24T22:19:54.864Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 140, in handler
[2024-03-24T22:19:54.864Z]     raise e from e
[2024-03-24T22:19:54.864Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2024-03-24T22:19:54.864Z]     return func(*args, **kwargs)
[2024-03-24T22:19:54.864Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 175, in handler
[2024-03-24T22:19:54.864Z]     return func(self, *args, **kwargs)
[2024-03-24T22:19:54.864Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 106, in handler
[2024-03-24T22:19:54.864Z]     raise MilvusException(
[2024-03-24T22:19:54.864Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>
[2024-03-24T22:19:54.864Z]  (api_request.py:45)
[2024-03-24T22:19:54.864Z] [2024-03-24 19:45:47 - ERROR - ci_test]: Traceback (most recent call last):
[2024-03-24T22:19:54.864Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-03-24T22:19:54.864Z]     res = func(*args, **_kwargs)
[2024-03-24T22:19:54.864Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-03-24T22:19:54.864Z]     return func(*arg, **kwargs)
[2024-03-24T22:19:54.864Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 1356, in create_index
[2024-03-24T22:19:54.864Z]     return conn.create_index(self._name, field_name, index_params, timeout=timeout, **kwargs)
[2024-03-24T22:19:54.864Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 140, in handler
[2024-03-24T22:19:54.864Z]     raise e from e
[2024-03-24T22:19:54.864Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2024-03-24T22:19:54.864Z]     return func(*args, **kwargs)
[2024-03-24T22:19:54.864Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 175, in handler
[2024-03-24T22:19:54.865Z]     return func(self, *args, **kwargs)
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 106, in handler
[2024-03-24T22:19:54.865Z]     raise MilvusException(
[2024-03-24T22:19:54.865Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>
[2024-03-24T22:19:54.865Z]  (api_request.py:45)
[2024-03-24T22:19:54.865Z] [2024-03-24 19:45:47 - ERROR - ci_test]: Traceback (most recent call last):
[2024-03-24T22:19:54.865Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-03-24T22:19:54.865Z]     res = func(*args, **_kwargs)
[2024-03-24T22:19:54.865Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-03-24T22:19:54.865Z]     return func(*arg, **kwargs)
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 1356, in create_index
[2024-03-24T22:19:54.865Z]     return conn.create_index(self._name, field_name, index_params, timeout=timeout, **kwargs)
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 140, in handler
[2024-03-24T22:19:54.865Z]     raise e from e
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2024-03-24T22:19:54.865Z]     return func(*args, **kwargs)
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 175, in handler
[2024-03-24T22:19:54.865Z]     return func(self, *args, **kwargs)
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 106, in handler
[2024-03-24T22:19:54.865Z]     raise MilvusException(
[2024-03-24T22:19:54.865Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>
[2024-03-24T22:19:54.865Z]  (api_request.py:45)
[2024-03-24T22:19:54.865Z] [2024-03-24 19:45:47 - ERROR - ci_test]: Traceback (most recent call last):
[2024-03-24T22:19:54.865Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-03-24T22:19:54.865Z]     res = func(*args, **_kwargs)
[2024-03-24T22:19:54.865Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-03-24T22:19:54.865Z]     return func(*arg, **kwargs)
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 1356, in create_index
[2024-03-24T22:19:54.865Z]     return conn.create_index(self._name, field_name, index_params, timeout=timeout, **kwargs)
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 140, in handler
[2024-03-24T22:19:54.865Z]     raise e from e
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2024-03-24T22:19:54.865Z]     return func(*args, **kwargs)
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 175, in handler
[2024-03-24T22:19:54.865Z]     return func(self, *args, **kwargs)
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 106, in handler
[2024-03-24T22:19:54.865Z]     raise MilvusException(
[2024-03-24T22:19:54.865Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>
[2024-03-24T22:19:54.865Z]  (api_request.py:45)
[2024-03-24T22:19:54.865Z] [2024-03-24 19:45:47 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)> (api_request.py:46)
[2024-03-24T22:19:54.865Z] [2024-03-24 19:45:47 - ERROR - ci_test]: Traceback (most recent call last):
[2024-03-24T22:19:54.865Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-03-24T22:19:54.865Z]     res = func(*args, **_kwargs)
[2024-03-24T22:19:54.865Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-03-24T22:19:54.865Z]     return func(*arg, **kwargs)
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 1356, in create_index
[2024-03-24T22:19:54.865Z]     return conn.create_index(self._name, field_name, index_params, timeout=timeout, **kwargs)
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 140, in handler
[2024-03-24T22:19:54.865Z]     raise e from e
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2024-03-24T22:19:54.865Z]     return func(*args, **kwargs)
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 175, in handler
[2024-03-24T22:19:54.865Z]     return func(self, *args, **kwargs)
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 106, in handler
[2024-03-24T22:19:54.865Z]     raise MilvusException(
[2024-03-24T22:19:54.865Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)>
[2024-03-24T22:19:54.865Z]  (api_request.py:45)
[2024-03-24T22:19:54.865Z] [2024-03-24 19:45:47 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)> (api_request.py:46)
[2024-03-24T22:19:54.865Z] [2024-03-24 19:45:47 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)> (api_request.py:46)
[2024-03-24T22:19:54.865Z] [2024-03-24 19:45:47 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)> (api_request.py:46)
[2024-03-24T22:19:54.865Z] [2024-03-24 19:45:47 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)> (api_request.py:46)
[2024-03-24T22:19:54.865Z] [2024-03-24 19:45:47 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=Retry timeout: 1200s, message=Retry timeout: 1200s, message=collection index_ueNllJHz create index  timeout in 1200s)> (api_request.py:46)
[2024-03-24T22:19:54.865Z] [2024-03-24 19:45:47 - ERROR - ci_test]: Traceback (most recent call last):
[2024-03-24T22:19:54.865Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/util_pymilvus.py", line 1051, in run
[2024-03-24T22:19:54.865Z]     super(MyThread, self).run()
[2024-03-24T22:19:54.865Z]   File "/usr/local/lib/python3.8/threading.py", line 870, in run
[2024-03-24T22:19:54.865Z]     self._target(*self._args, **self._kwargs)
[2024-03-24T22:19:54.865Z]   File "/home/jenkins/agent/workspace/tests/python_client/testcases/test_index.py", line 2193, in build
[2024-03-24T22:19:54.865Z]     collection_w.create_index(ct.default_float_vec_field_name, ct.default_diskann_index)
[2024-03-24T22:19:54.865Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/wrapper.py", line 33, in inner_wrapper
[2024-03-24T22:19:54.865Z]     res, result = func(*args, **kwargs)
[2024-03-24T22:19:54.865Z]   File "/home/jenkins/agent/workspace/tests/python_client/base/collection_wrapper.py", line 313, in create_index
[2024-03-24T22:19:54.865Z]     check_result = ResponseChecker(res, func_name, check_task, check_items, check, **kwargs).run()
[2024-03-24T22:19:54.865Z]   File "/home/jenkins/agent/workspace/tests/python_client/check/func_check.py", line 34, in run
[2024-03-24T22:19:54.865Z]     result = self.assert_succ(self.succ, True)
[2024-03-24T22:19:54.865Z]   File "/home/jenkins/agent/workspace/tests/python_client/check/func_check.py", line 112, in assert_succ
[2024-03-24T22:19:54.865Z]     assert actual is expect, f"Response of API {self.func_name} expect {expect}, but got {actual}"
[2024-03-24T22:19:54.865Z] AssertionError: Response of API create_index expect True, but got False
[2024-03-24T22:19:54.865Z]  (util_pymilvus.py:1054)

[xiaocai2333]

It seems that the task is stuck in IndexNode.

[xiaocai2333]

Compared with the normal case, it should be stuck in initializing S3 client.

[xiaocai2333]

The first failure was waiting for the index file to be uploaded, and it seemed to be stuck in the interaction with minio.

[xiaocai2333]

It is possible that the minio version is older(RELEASE.2021-02-14T04-01-33Z). Please use a newer version of minio to test. /assign @NicoYuan1986

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.