[Bug]: [benchmark] drop_collection failed to report an error: "etcdserver: request timed out"

[elstic]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version:milvus-io-2.2.5-053acd3-20230325
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):  kafka  
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task : fouramf-7jzhc case: test_concurrent_locust_100m_hnsw_ddl_dql_filter_kafka_cluster

server:

fouramf-7jzhc-15-5414-etcd-0                                     1/1     Running     0               2d14h   10.104.1.33    4am-node10   <none>           <none>
fouramf-7jzhc-15-5414-etcd-1                                     1/1     Running     0               2d14h   10.104.6.142   4am-node13   <none>           <none>
fouramf-7jzhc-15-5414-etcd-2                                     1/1     Running     0               2d14h   10.104.4.52    4am-node11   <none>           <none>
fouramf-7jzhc-15-5414-kafka-0                                    1/1     Running     1 (2d14h ago)   2d14h   10.104.1.35    4am-node10   <none>           <none>
fouramf-7jzhc-15-5414-kafka-1                                    1/1     Running     1 (2d14h ago)   2d14h   10.104.6.141   4am-node13   <none>           <none>
fouramf-7jzhc-15-5414-kafka-2                                    1/1     Running     0               2d14h   10.104.4.50    4am-node11   <none>           <none>
fouramf-7jzhc-15-5414-milvus-datacoord-8bf849497-mjsgm           1/1     Running     0               2d14h   10.104.9.20    4am-node14   <none>           <none>
fouramf-7jzhc-15-5414-milvus-datanode-7fb647c564-ms4sn           1/1     Running     0               2d14h   10.104.5.21    4am-node12   <none>           <none>
fouramf-7jzhc-15-5414-milvus-indexcoord-6c585d9b65-p2n9m         1/1     Running     0               2d14h   10.104.6.133   4am-node13   <none>           <none>
fouramf-7jzhc-15-5414-milvus-indexnode-5dc65d57b8-99ms8          1/1     Running     0               2d14h   10.104.5.22    4am-node12   <none>           <none>
fouramf-7jzhc-15-5414-milvus-proxy-6f8bbdd6bf-2jw9v              1/1     Running     0               2d14h   10.104.6.132   4am-node13   <none>           <none>
fouramf-7jzhc-15-5414-milvus-querycoord-68c649559d-sbg5q         1/1     Running     0               2d14h   10.104.4.44    4am-node11   <none>           <none>
fouramf-7jzhc-15-5414-milvus-querynode-f7f4bb7f6-fp977           1/1     Running     0               2d14h   10.104.6.134   4am-node13   <none>           <none>
fouramf-7jzhc-15-5414-milvus-querynode-f7f4bb7f6-xn6r8           1/1     Running     0               2d14h   10.104.9.21    4am-node14   <none>           <none>
fouramf-7jzhc-15-5414-milvus-rootcoord-779c5b458d-rpsqp          1/1     Running     0               2d14h   10.104.1.29    4am-node10   <none>           <none>
fouramf-7jzhc-15-5414-minio-0                                    1/1     Running     0               2d14h   10.104.9.24    4am-node14   <none>           <none>
fouramf-7jzhc-15-5414-minio-1                                    1/1     Running     0               2d14h   10.104.6.140   4am-node13   <none>           <none>
fouramf-7jzhc-15-5414-minio-2                                    1/1     Running     0               2d14h   10.104.4.51    4am-node11   <none>           <none>
fouramf-7jzhc-15-5414-minio-3                                    1/1     Running     0               2d14h   10.104.1.37    4am-node10   <none>           <none>
fouramf-7jzhc-15-5414-zookeeper-0                                1/1     Running     0               2d14h   10.104.1.34    4am-node10   <none>           <none>
fouramf-7jzhc-15-5414-zookeeper-1                                1/1     Running     0               2d14h   10.104.6.139   4am-node13   <none>           <none>
fouramf-7jzhc-15-5414-zookeeper-2                                1/1     Running     0               2d14h   10.104.4.49    4am-node11   <none>           <none>

client error log:

Expected Behavior

No response

Steps To Reproduce

1. create a collection or use an existing collection
        2. build index on vector column
        3. insert a certain number of vectors
        4. flush collection
        5. build index on vector column with the same parameters
        6. build index on on scalars column or not
        7. count the total number of rows
        8. load collection
        9. perform concurrent operations
        10. clean all collections or not

Milvus Log

No response

Anything else?

No response

[yanliang567]

sounds like an occasional etcd slow issue. lets see if it reproduces tonight.

[yanliang567]

@elstic does this keep reproducing recently?

/assign @elstic

[elstic]

@elstic does this keep reproducing recently?

/assign @elstic

Not every day, yesterday there was another .

argo task : platform-fouramf-cghx2

[yanliang567]

sounds like an occasional issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.