[Bug]: [benchmark][FLAT][cluster]milvus search failed and report:"_InactiveRpcError of RPC that terminated with"

[elstic]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version:2.2.0-20230408-dfee1062
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task : fouramf-cron-1680969600 , id : 1 case : test_recall_sift_flat_cluster

server

fouramf-cron-1680969600-1-etcd-0                                  1/1     Running     0               5m51s   10.104.6.223    4am-node13   <none>           <none>
fouramf-cron-1680969600-1-etcd-1                                  1/1     Running     0               5m51s   10.104.4.249    4am-node11   <none>           <none>
fouramf-cron-1680969600-1-etcd-2                                  1/1     Running     0               5m50s   10.104.9.163    4am-node14   <none>           <none>
fouramf-cron-1680969600-1-milvus-datacoord-8d8997b74-dm7t5        1/1     Running     1 (110s ago)    5m51s   10.104.13.177   4am-node16   <none>           <none>
fouramf-cron-1680969600-1-milvus-datanode-67485b5c44-kcqmj        1/1     Running     1 (110s ago)    5m51s   10.104.13.180   4am-node16   <none>           <none>
fouramf-cron-1680969600-1-milvus-indexcoord-5b67bd5f7d-b9zgl      1/1     Running     1 (110s ago)    5m51s   10.104.13.179   4am-node16   <none>           <none>
fouramf-cron-1680969600-1-milvus-indexnode-7c9964bbc5-wr64l       1/1     Running     0               5m51s   10.104.13.175   4am-node16   <none>           <none>
fouramf-cron-1680969600-1-milvus-proxy-6dc466c988-ztrdr           1/1     Running     1 (110s ago)    5m51s   10.104.13.178   4am-node16   <none>           <none>
fouramf-cron-1680969600-1-milvus-querycoord-746fc57589-tspwc      1/1     Running     1 (110s ago)    5m51s   10.104.12.195   4am-node17   <none>           <none>
fouramf-cron-1680969600-1-milvus-querynode-7cf7956cd9-nhvtg       1/1     Running     0               5m51s   10.104.12.194   4am-node17   <none>           <none>
fouramf-cron-1680969600-1-milvus-rootcoord-847b5bf6d7-t6l88       1/1     Running     1 (110s ago)    5m51s   10.104.13.176   4am-node16   <none>           <none>
fouramf-cron-1680969600-1-minio-0                                 1/1     Running     0               5m51s   10.104.9.158    4am-node14   <none>           <none>
fouramf-cron-1680969600-1-minio-1                                 1/1     Running     0               5m51s   10.104.4.247    4am-node11   <none>           <none>
fouramf-cron-1680969600-1-minio-2                                 1/1     Running     0               5m51s   10.104.6.221    4am-node13   <none>           <none>
fouramf-cron-1680969600-1-minio-3                                 1/1     Running     0               5m50s   10.104.5.35     4am-node12   <none>           <none>
fouramf-cron-1680969600-1-pulsar-bookie-0                         1/1     Running     0               5m51s   10.104.4.245    4am-node11   <none>           <none>
fouramf-cron-1680969600-1-pulsar-bookie-1                         1/1     Running     0               5m51s   10.104.9.161    4am-node14   <none>           <none>
fouramf-cron-1680969600-1-pulsar-bookie-2                         1/1     Running     0               5m51s   10.104.6.229    4am-node13   <none>           <none>
fouramf-cron-1680969600-1-pulsar-bookie-init-q4fzj                0/1     Completed   0               5m51s   10.104.4.242    4am-node11   <none>           <none>
fouramf-cron-1680969600-1-pulsar-broker-0                         1/1     Running     0               5m51s   10.104.6.217    4am-node13   <none>           <none>
fouramf-cron-1680969600-1-pulsar-proxy-0                          1/1     Running     0               5m51s   10.104.6.216    4am-node13   <none>           <none>
fouramf-cron-1680969600-1-pulsar-pulsar-init-lp8st                0/1     Completed   0               5m51s   10.104.4.241    4am-node11   <none>           <none>
fouramf-cron-1680969600-1-pulsar-recovery-0                       1/1     Running     0               5m51s   10.104.9.154    4am-node14   <none>           <none>
fouramf-cron-1680969600-1-pulsar-zookeeper-0                      1/1     Running     0               5m51s   10.104.9.157    4am-node14   <none>           <none>
fouramf-cron-1680969600-1-pulsar-zookeeper-1                      1/1     Running     0               5m14s   10.104.4.253    4am-node11   <none>           <none>
fouramf-cron-1680969600-1-pulsar-zookeeper-2                      1/1     Running     0               4m39s   10.104.1.128    4am-node10   <none>           <none>

client log: flat.log

Expected Behavior

No response

Steps To Reproduce

1. create a collection or use an existing collection
        2. insert training dataset
        3. flush collection
        4. clean index and build new index
        5. load collection
        6. search with different parameters
        7. clean all collections

Milvus Log

No response

Anything else?

No response

[yanliang567]

/assign @jiaoew1991 /unassign

[jiaoew1991]

/assign @yah01 /unassign

[elstic]

This issue did not reappear.

image: 2.2.0-20230604-29579aae

[elstic]

The issue no longer exists and I will close it. Verify version: 2.2.0-20230801-53a23fc8