milvus
milvus copied to clipboard
Published
20 hours ago •
milvus-io
milvus-io
[Bug]: [perf] Milvus search raise an error"failed to search, err: invalid args", when seach nq=1, topk=100
Is there an existing issue for this?
- [X] I have searched the existing issues
Environment
- Milvus version:master-20230509-893e31d1
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka): pulsar
- SDK version(e.g. pymilvus v2.0.0rc2):2.4.0dev12
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:
Current Behavior
release_name_prefix: perf-standalone-1-1683642600 deploy_config: fouramf-server-standalone-8c24m case_params: fouramf-client-search-nq1000-topk1000 other_params: -s --client_version=2.4.0 case_name: test_search_custom_parameters
server:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
perf-standalone42600-3-46-2582-etcd-0 1/1 Running 0 32m 10.104.18.63 4am-node25 <none> <none>
perf-standalone42600-3-46-2582-etcd-1 1/1 Running 0 32m 10.104.20.240 4am-node22 <none> <none>
perf-standalone42600-3-46-2582-etcd-2 1/1 Running 0 32m 10.104.22.128 4am-node26 <none> <none>
perf-standalone42600-3-46-2582-milvus-datacoord-bf866444c-p4p6k 1/1 Running 1 (28m ago) 32m 10.104.19.117 4am-node28 <none> <none>
perf-standalone42600-3-46-2582-milvus-datanode-55d9dbf87d-cjs6t 1/1 Running 1 (28m ago) 32m 10.104.19.118 4am-node28 <none> <none>
perf-standalone42600-3-46-2582-milvus-indexcoord-f89c6df58clbwl 1/1 Running 0 32m 10.104.16.60 4am-node21 <none> <none>
perf-standalone42600-3-46-2582-milvus-indexnode-7477cc456cpcj5x 1/1 Running 0 32m 10.104.4.12 4am-node11 <none> <none>
perf-standalone42600-3-46-2582-milvus-proxy-66749b98cd-tlm5s 1/1 Running 1 (28m ago) 32m 10.104.6.227 4am-node13 <none> <none>
perf-standalone42600-3-46-2582-milvus-querycoord-76566667452p9c 1/1 Running 1 (28m ago) 32m 10.104.18.55 4am-node25 <none> <none>
perf-standalone42600-3-46-2582-milvus-querynode-597cfd545cmmw77 1/1 Running 0 32m 10.104.1.222 4am-node10 <none> <none>
perf-standalone42600-3-46-2582-milvus-rootcoord-9d77cc775-cfcqr 1/1 Running 1 (28m ago) 32m 10.104.16.62 4am-node21 <none> <none>
perf-standalone42600-3-46-2582-minio-0 1/1 Running 0 32m 10.104.18.62 4am-node25 <none> <none>
perf-standalone42600-3-46-2582-minio-1 1/1 Running 0 32m 10.104.22.123 4am-node26 <none> <none>
perf-standalone42600-3-46-2582-minio-2 1/1 Running 0 32m 10.104.16.64 4am-node21 <none> <none>
perf-standalone42600-3-46-2582-minio-3 1/1 Running 0 32m 10.104.20.243 4am-node22 <none> <none>
perf-standalone42600-3-46-2582-pulsar-bookie-0 1/1 Running 0 32m 10.104.20.241 4am-node22 <none> <none>
perf-standalone42600-3-46-2582-pulsar-bookie-1 1/1 Running 0 32m 10.104.22.129 4am-node26 <none> <none>
perf-standalone42600-3-46-2582-pulsar-bookie-2 1/1 Running 0 32m 10.104.16.68 4am-node21 <none> <none>
perf-standalone42600-3-46-2582-pulsar-bookie-init-jqr2q 0/1 Completed 0 32m 10.104.18.53 4am-node25 <none> <none>
perf-standalone42600-3-46-2582-pulsar-broker-0 1/1 Running 0 32m 10.104.6.228 4am-node13 <none> <none>
perf-standalone42600-3-46-2582-pulsar-proxy-0 1/1 Running 0 32m 10.104.18.54 4am-node25 <none> <none>
perf-standalone42600-3-46-2582-pulsar-pulsar-init-g6kbf 0/1 Completed 0 32m 10.104.19.119 4am-node28 <none> <none>
perf-standalone42600-3-46-2582-pulsar-recovery-0 1/1 Running 0 32m 10.104.16.61 4am-node21 <none> <none>
perf-standalone42600-3-46-2582-pulsar-zookeeper-0 1/1 Running 0 32m 10.104.22.122 4am-node26 <none> <none>
perf-standalone42600-3-46-2582-pulsar-zookeeper-1 1/1 Running 0 32m 10.104.16.80 4am-node21 <none> <none>
perf-standalone42600-3-46-2582-pulsar-zookeeper-2 1/1 Running 0 29m 10.104.18.78 4am-node25 <none> <none>
client log:
[2023-05-09 15:03:54,589 - INFO - fouram]: [PerfTemplate] Actual parameters used: {'dataset_params': {'dim': 128, 'dataset_name': 'sift', 'dataset_size': '10m', 'ni_per': 50000, 'metric_type': 'L2', 'req_run_counts': 1000}, 'load_params': {}, 'search_params': {'nq': 1, 'param': {'metric_type': 'L2', 'params': {'ef': 16}}, 'top_k': 100, 'expr': None}, 'index_params': {'index_type': 'HNSW', 'index_param': {'M': 8, 'efConstruction': 200}}} (performance_template.py:60)
[2023-05-09 15:03:54,589 - INFO - fouram]: [Base] Params of search: nq:1, anns_field:float_vector, param:{'metric_type': 'L2', 'params': {'ef': 16}}, limit:100, expr:"None", kwargs:{} (base.py:338)
[2023-05-09 15:03:54,596 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)>, <Time:{'RPC start': '2023-05-09 15:03:54.589918', 'RPC error': '2023-05-09 15:03:54.596139'}> (decorators.py:108)
[2023-05-09 15:03:54,598 - ERROR - fouram]: Traceback (most recent call last):
File "/src/fouram/client/util/api_request.py", line 33, in inner_wrapper
res = func(*args, **kwargs)
File "/src/fouram/client/util/api_request.py", line 70, in api_request
return func(*arg, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 660, in search
res = conn.search(self._name, data, anns_field, param, limit, expr,
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 109, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 105, in handler
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 136, in handler
ret = func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 85, in handler
raise e
File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 50, in handler
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 518, in search
return self._execute_search_requests(requests, timeout, round_decimal=round_decimal, auto_id=auto_id, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 487, in _execute_search_requests
raise pre_err
File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 478, in _execute_search_requests
raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)>
(api_request.py:48)
Expected Behavior
No response
Steps To Reproduce
1. create a collection
2. build index on vector column
3. insert 10m of vectors
4. flush collection
5. build index on vector column with the same parameters
6. count the total number of rows
7. load collection
8. search collection with different parameters
Milvus Log
No response
Anything else?
[2023-05-09 15:03:54,589 - INFO - fouram]: [PerfTemplate] Actual parameters used: {'dataset_params': {'dim': 128, 'dataset_name': 'sift', 'dataset_size': '10m', 'ni_per': 50000, 'metric_type': 'L2', 'req_run_counts': 1000}, 'load_params': {}, 'search_params': {'nq': 1, 'param': {'metric_type': 'L2', 'params': {'ef': 16}}, 'top_k': 100, 'expr': None}, 'index_params': {'index_type': 'HNSW', 'index_param': {'M': 8, 'efConstruction': 200}}} (performance_template.py:60)
@liliu-z @cydrain Why user can not get full error message? The error message should not be only "failed to search, err: invalid args" but the main reason
image：master-20230515-30415e1b
release_name_prefix： perf-standalone-1-1684420200 deploy_config： fouramf-server-standalone-8c24m case_params： fouramf-client-search-nq1000-topk1000-scalar other_params： -s --client_version=2.4.0 case_name： test_search_custom_parameters
server；
perf-standalone20200-4-5-1516-etcd-0 1/1 Running 0 32m 10.104.19.132 4am-node28 <none> <none>
perf-standalone20200-4-5-1516-etcd-1 1/1 Running 0 32m 10.104.22.166 4am-node26 <none> <none>
perf-standalone20200-4-5-1516-etcd-2 1/1 Running 0 32m 10.104.17.11 4am-node23 <none> <none>
perf-standalone20200-4-5-1516-milvus-datacoord-fd5c9779f-7czlf 1/1 Running 1 (28m ago) 32m 10.104.1.230 4am-node10 <none> <none>
perf-standalone20200-4-5-1516-milvus-datanode-7b56c7856-mnmfx 1/1 Running 1 (28m ago) 32m 10.104.5.146 4am-node12 <none> <none>
perf-standalone20200-4-5-1516-milvus-indexcoord-577d6d4f984tpq8 1/1 Running 0 32m 10.104.20.32 4am-node22 <none> <none>
perf-standalone20200-4-5-1516-milvus-indexnode-755f54b4b5-h7wqq 1/1 Running 0 32m 10.104.1.231 4am-node10 <none> <none>
perf-standalone20200-4-5-1516-milvus-proxy-5878b6b96-mws8w 1/1 Running 1 (28m ago) 32m 10.104.19.116 4am-node28 <none> <none>
perf-standalone20200-4-5-1516-milvus-querycoord-64d4d55457j29d7 1/1 Running 1 (28m ago) 32m 10.104.20.33 4am-node22 <none> <none>
perf-standalone20200-4-5-1516-milvus-querynode-5774864847-dwbv7 1/1 Running 0 32m 10.104.18.95 4am-node25 <none> <none>
perf-standalone20200-4-5-1516-milvus-rootcoord-6bcf454479-g8bjd 1/1 Running 1 (28m ago) 32m 10.104.20.35 4am-node22 <none> <none>
perf-standalone20200-4-5-1516-minio-0 1/1 Running 0 32m 10.104.20.47 4am-node22 <none> <none>
perf-standalone20200-4-5-1516-minio-1 1/1 Running 0 32m 10.104.1.241 4am-node10 <none> <none>
perf-standalone20200-4-5-1516-minio-2 1/1 Running 0 32m 10.104.22.165 4am-node26 <none> <none>
perf-standalone20200-4-5-1516-minio-3 1/1 Running 0 32m 10.104.19.139 4am-node28 <none> <none>
perf-standalone20200-4-5-1516-pulsar-bookie-0 1/1 Running 0 32m 10.104.19.131 4am-node28 <none> <none>
perf-standalone20200-4-5-1516-pulsar-bookie-1 1/1 Running 0 32m 10.104.17.5 4am-node23 <none> <none>
perf-standalone20200-4-5-1516-pulsar-bookie-2 1/1 Running 0 32m 10.104.22.169 4am-node26 <none> <none>
perf-standalone20200-4-5-1516-pulsar-bookie-init-f8272 0/1 Completed 0 32m 10.104.17.242 4am-node23 <none> <none>
perf-standalone20200-4-5-1516-pulsar-broker-0 1/1 Running 0 32m 10.104.20.36 4am-node22 <none> <none>
perf-standalone20200-4-5-1516-pulsar-proxy-0 1/1 Running 0 32m 10.104.1.229 4am-node10 <none> <none>
perf-standalone20200-4-5-1516-pulsar-pulsar-init-wnm4m 0/1 Completed 0 32m 10.104.20.34 4am-node22 <none> <none>
perf-standalone20200-4-5-1516-pulsar-recovery-0 1/1 Running 0 32m 10.104.17.243 4am-node23 <none> <none>
perf-standalone20200-4-5-1516-pulsar-zookeeper-0 1/1 Running 0 32m 10.104.20.46 4am-node22 <none> <none>
perf-standalone20200-4-5-1516-pulsar-zookeeper-1 1/1 Running 0 31m 10.104.6.219 4am-node13 <none> <none>
perf-standalone20200-4-5-1516-pulsar-zookeeper-2 1/1 Running 0 28m 10.104.19.148 4am-node28 <none> <none>
client log：
[2023-05-18 15:03:24,189 - INFO - fouram]: [PerfTemplate] Actual parameters used: {'dataset_params': {'metric_type': 'L2', 'dim': 128, 'dataset_name': 'sift', 'dataset_size': '10m', 'ni_per': 50000, 'req_run_counts': 1000}, 'collection_params': {'other_fields': ['int64_1', 'float_1', 'double_1', 'varchar_1']}, 'load_params': {}, 'query_params': {}, 'search_params': {'nq': 1, 'param': {'metric_type': 'L2', 'params': {'ef': 16}}, 'top_k': 100, 'expr': None}, 'index_params': {'index_type': 'HNSW', 'index_param': {'M': 8, 'efConstruction': 200}}} (performance_template.py:63)
[2023-05-18 15:03:24,189 - INFO - fouram]: [Base] Params of search: nq:1, anns_field:float_vector, param:{'metric_type': 'L2', 'params': {'ef': 16}}, limit:100, expr:"None", kwargs:{} (base.py:482)
[2023-05-18 15:03:24,196 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)>, <Time:{'RPC start': '2023-05-18 15:03:24.189427', 'RPC error': '2023-05-18 15:03:24.196826'}> (decorators.py:108)
[2023-05-18 15:03:24,199 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)> (api_request.py:49)
[2023-05-18 15:03:24,199 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)> (func_check.py:52)
[2023-05-18 15:03:24,199 - ERROR - fouram]: [Search] Search raise error: (common_cases.py:602)
[2023-05-18 15:03:24,199 - INFO - fouram]: [PerfTemplate] Actual parameters used: {'dataset_params': {'metric_type': 'L2', 'dim': 128, 'dataset_name': 'sift', 'dataset_size': '10m', 'ni_per': 50000, 'req_run_counts': 1000}, 'collection_params': {'other_fields': ['int64_1', 'float_1', 'double_1', 'varchar_1']}, 'load_params': {}, 'query_params': {}, 'search_params': {'nq': 100, 'param': {'metric_type': 'L2', 'params': {'ef': 16}}, 'top_k': 100, 'expr': None}, 'index_params': {'index_type': 'HNSW', 'index_param': {'M': 8, 'efConstruction': 200}}} (performance_template.py:63)
[2023-05-18 15:03:24,199 - INFO - fouram]: [Base] Params of search: nq:100, anns_field:float_vector, param:{'metric_type': 'L2', 'params': {'ef': 16}}, limit:100, expr:"None", kwargs:{} (base.py:482)
[2023-05-18 15:03:24,214 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)>, <Time:{'RPC start': '2023-05-18 15:03:24.206608', 'RPC error': '2023-05-18 15:03:24.214798'}> (decorators.py:108)
[2023-05-18 15:03:24,215 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)> (api_request.py:49)
[2023-05-18 15:03:24,215 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)> (func_check.py:52)
[2023-05-18 15:03:24,215 - ERROR - fouram]: [Search] Search raise error: (common_cases.py:602)
[2023-05-18 15:03:24,215 - INFO - fouram]: [PerfTemplate] Actual parameters used: {'dataset_params': {'metric_type': 'L2', 'dim': 128, 'dataset_name': 'sift', 'dataset_size': '10m', 'ni_per': 50000, 'req_run_counts': 1000}, 'collection_params': {'other_fields': ['int64_1', 'float_1', 'double_1', 'varchar_1']}, 'load_params': {}, 'query_params': {}, 'search_params': {'nq': 1000, 'param': {'metric_type': 'L2', 'params': {'ef': 16}}, 'top_k': 100, 'expr': None}, 'index_params': {'index_type': 'HNSW', 'index_param': {'M': 8, 'efConstruction': 200}}} (performance_template.py:63)
[2023-05-18 15:03:24,215 - INFO - fouram]: [Base] Params of search: nq:1000, anns_field:float_vector, param:{'metric_type': 'L2', 'params': {'ef': 16}}, limit:100, expr:"None", kwargs:{} (base.py:482)
[2023-05-18 15:03:24,292 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)>, <Time:{'RPC start': '2023-05-18 15:03:24.277648', 'RPC error': '2023-05-18 15:03:24.292355'}> (decorators.py:108)
[2023-05-18 15:03:24,292 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)> (api_request.py:49)
[2023-05-18 15:03:24,292 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)> (func_check.py:52)
[2023-05-18 15:03:24,292 - ERROR - fouram]: [Search] Search raise error: (common_cases.py:602)
[2023-05-18 15:03:24,292 - INFO - fouram]: [PerfTemplate] Actual parameters used: {'dataset_params': {'metric_type': 'L2', 'dim': 128, 'dataset_name': 'sift', 'dataset_size': '10m', 'ni_per': 50000, 'req_run_counts': 1000}, 'collection_params': {'other_fields': ['int64_1', 'float_1', 'double_1', 'varchar_1']}, 'load_params': {}, 'query_params': {}, 'search_params': {'nq': 1, 'param': {'metric_type': 'L2', 'params': {'ef': 16}}, 'top_k': 1000, 'expr': None}, 'index_params': {'index_type': 'HNSW', 'index_param': {'M': 8, 'efConstruction': 200}}} (performance_template.py:63)
[2023-05-18 15:03:24,293 - INFO - fouram]: [Base] Params of search: nq:1, anns_field:float_vector, param:{'metric_type': 'L2', 'params': {'ef': 16}}, limit:1000, expr:"None", kwargs:{} (base.py:482)
[2023-05-18 15:03:24,299 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)>, <Time:{'RPC start': '2023-05-18 15:03:24.293217', 'RPC error': '2023-05-18 15:03:24.299496'}> (decorators.py:108)
[2023-05-18 15:03:24,299 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)> (api_request.py:49)
[2023-05-18 15:03:24,299 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)> (func_check.py:52)
[2023-05-18 15:03:24,299 - ERROR - fouram]: [Search] Search raise error: (common_cases.py:602)
[2023-05-18 15:03:24,300 - INFO - fouram]: [PerfTemplate] Actual parameters used: {'dataset_params': {'metric_type': 'L2', 'dim': 128, 'dataset_name': 'sift', 'dataset_size': '10m', 'ni_per': 50000, 'req_run_counts': 1000}, 'collection_params': {'other_fields': ['int64_1', 'float_1', 'double_1', 'varchar_1']}, 'load_params': {}, 'query_params': {}, 'search_params': {'nq': 100, 'param': {'metric_type': 'L2', 'params': {'ef': 16}}, 'top_k': 1000, 'expr': None}, 'index_params': {'index_type': 'HNSW', 'index_param': {'M': 8, 'efConstruction': 200}}} (performance_template.py:63)
[2023-05-18 15:03:24,300 - INFO - fouram]: [Base] Params of search: nq:100, anns_field:float_vector, param:{'metric_type': 'L2', 'params': {'ef': 16}}, limit:1000, expr:"None", kwargs:{} (base.py:482)
[2023-05-18 15:03:24,314 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)>, <Time:{'RPC start': '2023-05-18 15:03:24.306409', 'RPC error': '2023-05-18 15:03:24.314037'}> (decorators.py:108)
[2023-05-18 15:03:24,314 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)> (api_request.py:49)
[2023-05-18 15:03:24,314 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)> (func_check.py:52)
[2023-05-18 15:03:24,314 - ERROR - fouram]: [Search] Search raise error: (common_cases.py:602)
[2023-05-18 15:03:24,314 - INFO - fouram]: [PerfTemplate] Actual parameters used: {'dataset_params': {'metric_type': 'L2', 'dim': 128, 'dataset_name': 'sift', 'dataset_size': '10m', 'ni_per': 50000, 'req_run_counts': 1000}, 'collection_params': {'other_fields': ['int64_1', 'float_1', 'double_1', 'varchar_1']}, 'load_params': {}, 'query_params': {}, 'search_params': {'nq': 1000, 'param': {'metric_type': 'L2', 'params': {'ef': 16}}, 'top_k': 1000, 'expr': None}, 'index_params': {'index_type': 'HNSW', 'index_param': {'M': 8, 'efConstruction': 200}}} (performance_template.py:63)
[2023-05-18 15:03:24,314 - INFO - fouram]: [Base] Params of search: nq:1000, anns_field:float_vector, param:{'metric_type': 'L2', 'params': {'ef': 16}}, limit:1000, expr:"None", kwargs:{} (base.py:482)
[2023-05-18 15:03:24,387 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)>, <Time:{'RPC start': '2023-05-18 15:03:24.374510', 'RPC error': '2023-05-18 15:03:24.387688'}> (decorators.py:108)
[2023-05-18 15:03:24,388 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)> (api_request.py:49)
[2023-05-18 15:03:24,388 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=worker(1) query failed: [UnexpectedError] Assert "failed to search, err: invalid args" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:130
)> (func_check.py:52)
image: 2.2.0-20230612-ae2fe478
release_name_prefix
perf-standalone-1-1686580200
deploy_config
fouramf-server-standalone-8c24m
case_params
fouramf-client-search-nq1000-topk1000
other_params
-s --client_version=2.4.0
execut_type
-k
case_name
test_search_custom_parameters
server:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
perf-standalone80200-3-15-5281-etcd-0 1/1 Running 0 5m53s 10.104.4.26 4am-node11 <none> <none>
perf-standalone80200-3-15-5281-etcd-1 1/1 Running 0 5m53s 10.104.9.212 4am-node14 <none> <none>
perf-standalone80200-3-15-5281-etcd-2 1/1 Running 0 5m52s 10.104.6.170 4am-node13 <none> <none>
perf-standalone80200-3-15-5281-milvus-datacoord-779f6679b5n98x4 1/1 Running 1 (112s ago) 5m53s 10.104.17.170 4am-node23 <none> <none>
perf-standalone80200-3-15-5281-milvus-datanode-5f789cc777-z4645 1/1 Running 1 (113s ago) 5m53s 10.104.15.19 4am-node20 <none> <none>
perf-standalone80200-3-15-5281-milvus-indexcoord-5c847bb4cpzwsk 1/1 Running 1 (113s ago) 5m53s 10.104.15.18 4am-node20 <none> <none>
perf-standalone80200-3-15-5281-milvus-indexnode-7864b54bb4p2rbw 1/1 Running 0 5m53s 10.104.24.56 4am-node29 <none> <none>
perf-standalone80200-3-15-5281-milvus-proxy-b984554d5-f7dwx 1/1 Running 1 (112s ago) 5m53s 10.104.17.169 4am-node23 <none> <none>
perf-standalone80200-3-15-5281-milvus-querycoord-774887bdb5kz5r 1/1 Running 1 (112s ago) 5m53s 10.104.17.171 4am-node23 <none> <none>
perf-standalone80200-3-15-5281-milvus-querynode-6545d8767dwmbqx 1/1 Running 0 5m53s 10.104.17.172 4am-node23 <none> <none>
perf-standalone80200-3-15-5281-milvus-rootcoord-65b775cb54w7df4 1/1 Running 1 (113s ago) 5m53s 10.104.24.55 4am-node29 <none> <none>
perf-standalone80200-3-15-5281-minio-0 1/1 Running 0 5m53s 10.104.4.25 4am-node11 <none> <none>
perf-standalone80200-3-15-5281-minio-1 1/1 Running 0 5m53s 10.104.9.214 4am-node14 <none> <none>
perf-standalone80200-3-15-5281-minio-2 1/1 Running 0 5m52s 10.104.21.138 4am-node24 <none> <none>
perf-standalone80200-3-15-5281-minio-3 1/1 Running 0 5m52s 10.104.6.171 4am-node13 <none> <none>
perf-standalone80200-3-15-5281-pulsar-bookie-0 1/1 Running 0 5m53s 10.104.6.167 4am-node13 <none> <none>
perf-standalone80200-3-15-5281-pulsar-bookie-1 1/1 Running 0 5m52s 10.104.9.215 4am-node14 <none> <none>
perf-standalone80200-3-15-5281-pulsar-bookie-2 1/1 Running 0 5m51s 10.104.4.29 4am-node11 <none> <none>
perf-standalone80200-3-15-5281-pulsar-bookie-init-m8gpt 0/1 Completed 0 5m53s 10.104.4.15 4am-node11 <none> <none>
perf-standalone80200-3-15-5281-pulsar-broker-0 1/1 Running 0 5m53s 10.104.4.16 4am-node11 <none> <none>
perf-standalone80200-3-15-5281-pulsar-proxy-0 1/1 Running 0 5m52s 10.104.9.207 4am-node14 <none> <none>
perf-standalone80200-3-15-5281-pulsar-pulsar-init-vrv9p 0/1 Completed 0 5m53s 10.104.4.13 4am-node11 <none> <none>
perf-standalone80200-3-15-5281-pulsar-recovery-0 1/1 Running 0 5m53s 10.104.6.144 4am-node13 <none> <none>
perf-standalone80200-3-15-5281-pulsar-zookeeper-0 1/1 Running 0 5m53s 10.104.4.24 4am-node11 <none> <none>
perf-standalone80200-3-15-5281-pulsar-zookeeper-1 1/1 Running 0 4m20s 10.104.1.14 4am-node10 <none> <none>
perf-standalone80200-3-15-5281-pulsar-zookeeper-2 1/1 Running 0 3m18s 10.104.5.37 4am-node12 <none> <none>
log:
[2023-06-12 14:59:42,601 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=fail to search on all shard leaders, err=All attempts results:
attempt #1:code: UnexpectedError, error: fail to Search, QueryNode ID=3, reason=Search 3 failed, reason [UnexpectedError] Assert "false" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:169
=> Error in void knowhere::CheckIntegerRange(const Config&, const string&, int64_t, int64_t) at ConfAdapter.cpp:58: Param 'ef'(16) is not in range [100, 9223372036854775807] err %!w(<nil>)
attempt #2:context canceled
)>, <Time:{'RPC start': '2023-06-12 14:59:42.597826', 'RPC error': '2023-06-12 14:59:42.601744'}> (decorators.py:108)
[2023-06-12 14:59:42,604 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=fail to search on all shard leaders, err=All attempts results:
attempt #1:code: UnexpectedError, error: fail to Search, QueryNode ID=3, reason=Search 3 failed, reason [UnexpectedError] Assert "false" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:169
=> Error in void knowhere::CheckIntegerRange(const Config&, const string&, int64_t, int64_t) at ConfAdapter.cpp:58: Param 'ef'(16) is not in range [100, 9223372036854775807] err %!w(<nil>)
attempt #2:context canceled
)> (api_request.py:49)
[2023-06-12 14:59:42,604 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=fail to search on all shard leaders, err=All attempts results:
attempt #1:code: UnexpectedError, error: fail to Search, QueryNode ID=3, reason=Search 3 failed, reason [UnexpectedError] Assert "false" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:169
=> Error in void knowhere::CheckIntegerRange(const Config&, const string&, int64_t, int64_t) at ConfAdapter.cpp:58: Param 'ef'(16) is not in range [100, 9223372036854775807] err %!w(<nil>)
attempt #2:context canceled
)> (func_check.py:52)
[2023-06-12 14:59:42,605 - ERROR - fouram]: [Search] Search raise error: (common_cases.py:602)
[2023-06-12 14:59:42,605 - INFO - fouram]: [PerfTemplate] Actual parameters used: {'dataset_params': {'metric_type': 'L2', 'dim': 128, 'dataset_name': 'sift', 'dataset_size': '10m', 'ni_per': 50000, 'req_run_counts': 1000}, 'load_params': {}, 'query_params': {}, 'search_params': {'nq': 100, 'param': {'metric_type': 'L2', 'params': {'ef': 16}}, 'top_k': 100, 'expr': None}, 'index_params': {'index_type': 'HNSW', 'index_param': {'M': 8, 'efConstruction': 200}}} (performance_template.py:63)
[2023-06-12 14:59:42,605 - INFO - fouram]: [Base] Params of search: nq:100, anns_field:float_vector, param:{'metric_type': 'L2', 'params': {'ef': 16}}, limit:100, expr:"None", kwargs:{} (base.py:482)
[2023-06-12 14:59:42,617 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=fail to search on all shard leaders, err=All attempts results:
attempt #1:code: UnexpectedError, error: fail to Search, QueryNode ID=3, reason=Search 3 failed, reason [UnexpectedError] Assert "false" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:169
=> Error in void knowhere::CheckIntegerRange(const Config&, const string&, int64_t, int64_t) at ConfAdapter.cpp:58: Param 'ef'(16) is not in range [100, 9223372036854775807] err %!w(<nil>)
attempt #2:context canceled
)>, <Time:{'RPC start': '2023-06-12 14:59:42.612658', 'RPC error': '2023-06-12 14:59:42.617188'}> (decorators.py:108)
[2023-06-12 14:59:42,617 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=fail to search on all shard leaders, err=All attempts results:
attempt #1:code: UnexpectedError, error: fail to Search, QueryNode ID=3, reason=Search 3 failed, reason [UnexpectedError] Assert "false" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:169
=> Error in void knowhere::CheckIntegerRange(const Config&, const string&, int64_t, int64_t) at ConfAdapter.cpp:58: Param 'ef'(16) is not in range [100, 9223372036854775807] err %!w(<nil>)
attempt #2:context canceled
)> (api_request.py:49)
[2023-06-12 14:59:42,617 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=fail to search on all shard leaders, err=All attempts results:
attempt #1:code: UnexpectedError, error: fail to Search, QueryNode ID=3, reason=Search 3 failed, reason [UnexpectedError] Assert "false" at /go/src/github.com/milvus-io/milvus/internal/core/src/index/VectorMemIndex.cpp:169
=> Error in void knowhere::CheckIntegerRange(const Config&, const string&, int64_t, int64_t) at ConfAdapter.cpp:58: Param 'ef'(16) is not in range [100, 9223372036854775807] err %!w(<nil>)
attempt #2:context canceled
)> (func_check.py:52)
This issue requires that ef must be greater than topk, otherwise an error will be reported.
2.2.0-20230616-de66c8b5 release_name_prefix perf-standalone-1-1687098600 deploy_config fouramf-server-standalone-8c24m case_params fouramf-client-search-nq1000-topk1000
what is ef, how can i edit-ef
@booksource1 ef is a search parameter if you are running HNSW index:https://milvus.io/docs/index.md#HNSW