[Bug]: [perf] Milvus' search qps in gist's dataset decreases by about half

[jingkl]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version:
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Milvus search qps's recent week of testing is about half the size of the original 2.2.6 release The leftmost version is 2.2.6-41d9ab3d, The image tested in the figure below is date+commitid

gist-768d-hsnw index

gist-768d-ivf-flat index

gist-768d-diskann index

Expected Behavior

No response

Steps To Reproduce

1. create a collection 
        2. build index on vector column
        3. insert a 1m gist768d of vectors
        4. flush collection
        5. build index on vector column with the same parameters
        6. count the total number of rows
        7. load collection
        8. hnsw, ivf_flat concurrent [1, 20, 50, 100] search , diskann  concurrent [1, 20] search

Milvus Log

No response

Anything else?

No response

[yanliang567]

/assign @liliu-z it only reproduces on gist dataset. /unassign

[liliu-z]

Looks like this drop happened between f9d6ece and 41d9ab, and the corresponding Knowhere version is 1.3.12 and 1.3.11 respectively

[liliu-z]

/assign @hhy3

[liliu-z]

@jingkl can you help make sure we didn't use debug image for this test? This big jump looks scaring to me cc @yanliang567

[hhy3]

Perhaps I can not reproduce the results in the figure. Could you please provide more info like milvus configuration, hardware environment, and testing script?

[liliu-z]

/assign @jiaoew1991

[jiaoew1991]

/assign @yah01 It seems that some PRs fixed the bugs on April 30th. 🤔

[yah01]

The MaxNQ config item has been increased, will bench on it

[yah01]

Nothing changes after modified the MaxNQ back to 1000

[yah01]

/assign @zhagnlu

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[yanliang567]

@jingkl does it reproduce recently?

[jingkl]

2.2.14 search performance back to normal, close issue