[Bug]: [benchmark] insert 100 million data, 2 replica , concurrent operations, IVF_SQ8 index search latency is twice that of November 2022

[elstic]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version: 2.2.0-20230427-c82ec0b1
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Same scenario, insert 100 million data, concurrent operations, IVF_SQ8 index search latency is twice as high as in November 2022 .

image: 2.2.0-20230427-c82ec0b1 search latency : about 0.233s

image : 2.2.0-20221118-e7429f88 search latency : about 0.109s

Expected Behavior

No response

Steps To Reproduce

1. create a collection or use an existing collection
        2. build index on vector column
        3. insert a certain number of vectors
        4. flush collection
        5. build index on vector column with the same parameters
        6. build index on on scalars column or not
        7. count the total number of rows
        8. load collection
        9. perform concurrent operations (search、load、query、ddl)
        10. clean all collections or not

Milvus Log

No response

Anything else?

No response

[yanliang567]

/assign @MrPresent-Han /unassign

[xiaofan-luan]

/assign @cydrain

[xiaofan-luan]

@cydrain pls help on investigate it

[cydrain]

image: 2.2.0-20230427-c82ec0b1 (knowhere-1.3.12)

image : 2.2.0-20221118-e7429f88 (knowhere-1.3.2)

test with sift1m dataset, see no obvious performance difference

[elstic]

case :test_concurrent_locust_100m_ivf_sq8_ddl_dql_filter_replica2_cluster using dataset: SIFT , metirc_type:L2

search (weight=20):

 nq=10, top_k=10, search_param={"nprobe": 16},
 expr=eval("{'float_1': {'GT': -1.0, 'LT': parser_data_size(data_size) * 0.5}}")),

query (weight=10):

ids=[i for i in range(10)]

load(weight=1) ddl (weight=2)

create collection,insert vectors, flush, build index ,time.sleep(59), drop collection 

[elstic]

Synchronized information: ddl operations have slowed down more ： between 112s and 114s

In the previous version v2.2.4:

case: test_concurrent_locust_100m_ivf_sq8_ddl_dql_filter_replica2_cluster

[elstic]

/assign @jiaoew1991 ddl is slowing down, please help troubleshoot

[cydrain]

/unassign

[jiaoew1991]

Is this still a problem, @elstic?

[elstic]

Is this still a problem, @elstic?

This issue no longer exists. Please see the search latency of this case recently run, it and our benchmark data has been about the same, according to my judgment, should be this issue under the pr repair: https://github.com/milvus-io/milvus/issues/24022