[Bug]: [benchmark][standalone] Milvus search failed,raise an error"fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=stream operation failed: deadline exceed"

[jingkl]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version:master-20220920-5143e2a7
- Deployment mode(standalone or cluster):standalone
- SDK version(e.g. pymilvus v2.0.0rc2):2.2.0dev32
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

server-instance fouram-w5ccc-1 server-configmap server-single-8c16m client-configmap client-acc-sift-flat

sever:

NAME                                                              READY   STATUS      RESTARTS      AGE     IP             NODE         NOMINATED NODE   READINESS GATES
fouram-w5ccc-1-etcd-0                                             1/1     Running     0             112s    10.104.5.82    4am-node12   <none>           <none>
fouram-w5ccc-1-milvus-standalone-79c595d867-bvc6h                 1/1     Running     0             112s    10.104.6.121   4am-node13   <none>           <none>
fouram-w5ccc-1-minio-65968fd9-dh5nf                               1/1     Running     0             112s    10.104.6.120   4am-node13   <none>           <none>

client log:

2022-09-20 03:33:52,111] [    INFO] - Start load collection: sift_128_euclidean (milvus_benchmark.runners.accuracy:276)
[2022-09-20 03:33:57,188] [   DEBUG] - Milvus load_collection run in 5.0775s (milvus_benchmark.client:57)
[2022-09-20 03:33:57,188] [    INFO] - End load collection: sift_128_euclidean (milvus_benchmark.runners.accuracy:279)
[2022-09-20 03:33:57,189] [   DEBUG] - Milvus get_query_segment_info run in 0.0008s (milvus_benchmark.client:57)
[2022-09-20 03:33:57,189] [   DEBUG] - query_segment_info: [] (milvus_benchmark.runners.accuracy:282)
[2022-09-20 03:33:57,191] [    INFO] - Start run case (milvus_benchmark.main:88)
[2022-09-20 03:33:58,088] [   ERROR] - RPC error: [search], <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=stream operation failed: deadline exceed)>, <Time:{'RPC start': '2022-09-20 03:33:57.191366', 'RPC error': '2022-09-20 03:33:58.088386'}> (pymilvus.decorators:112)
[2022-09-20 03:33:58,089] [   ERROR] - Traceback (most recent call last):
  File "main.py", line 95, in run_suite
    result = runner.run_case(case_metric, **case)
  File "/src/milvus_benchmark/runners/accuracy.py", line 292, in run_case
    self.milvus.query(case_param["vector_query"], filter_query=case_param["filter_query"],
  File "/src/milvus_benchmark/client.py", line 53, in wrapper
    result = func(*args, **kwargs)
  File "/src/milvus_benchmark/client.py", line 346, in query
    result = self._milvus.search(tmp_collection_name, **params)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/stub.py", line 844, in search
    return handler.search(collection_name, data, anns_field, param, limit, expression,
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 113, in handler
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 109, in handler
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 139, in handler
    ret = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 89, in handler
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 51, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 472, in search
    return self._execute_search_requests(requests, timeout, **_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 436, in _execute_search_requests
    raise pre_err
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 427, in _execute_search_requests
    raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=stream operation failed: deadline exceed)>
 (milvus_benchmark.main:98)

Expected Behavior

No response

Steps To Reproduce

1.create an collection
2.insert sift1m data
3.create flat index, then  load
4.search raise error

Milvus Log

No response

Anything else?

client-acc-sift-flat:

{
	"config.yaml": "ann_accuracy:
		  collections:
		    -
		      milvus:
		        cache_config.cpu_cache_capacity: 16GB
		        engine_config.use_blas_threshold: 1100
		      server:
		        cpus: 12
		      source_file: /test/milvus/ann_hdf5/sift-128-euclidean.hdf5
		      collection_name: sift_128_euclidean
		      index_types: ['flat']
		      index_params:
		        nlist: [1024]
		      top_ks: [10]
		      nqs: [10000]
		      search_params:
		        nprobe: [1]
		"
}

[yanliang567]

/assign @jiaoew1991 /unassign

[congqixia]

Fix PR #19249 is merged. Could you please verify? @jingkl /unassign @jiaoew1991 /assign @jingkl

[wangting0128]

verify image：master-20221012-a8a07416