milvus icon indicating copy to clipboard operation
milvus copied to clipboard

[Bug]: [benchmark]load memory usage more

Open elstic opened this issue 1 year ago • 5 comments

Is there an existing issue for this?

  • [X] I have searched the existing issues

Environment

- Milvus version:
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Save load memory pr After entering, load memory usage is higher instead . index : HNSW dataset : 6m 128d image :

Adjusting memory usage before load:6G ,image : 2.2.0-20230324-9c3c29db image

Adjusting memory usage after load: 6.68G, image: xige-16-opt-mem-copy-aa2886b-20230403

image

Expected Behavior

Adjusted memory usage after load less than peak memory usage before load adjustment

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

elstic avatar Apr 04 '23 06:04 elstic

[cluster] load memory usage more

Current Behavior

Adjusting memory usage before load:4.3G (querynode) image

Adjusting memory usage after load: 5.5G (querynode) image

elstic avatar Apr 04 '23 06:04 elstic

/assign @xige-16

xiaofan-luan avatar Apr 04 '23 20:04 xiaofan-luan

/unassign

yanliang567 avatar Apr 05 '23 00:04 yanliang567

@xige-16 what is the progress now?

binbinlv avatar Apr 17 '23 02:04 binbinlv

2.2.0 has been verifyed , and can load more data

xige-16 avatar Apr 25 '23 06:04 xige-16

issue Validated, please be aware that: @yanliang567

After entering load pr, slightly more memory resources are needed when memory is sufficient, but more data can be loaded when resources are limited. See the table for details:

(standalone , index : HNSW)

data_size\image memory v2.2.5 2.2.0-20230425-e820479b
6m Sufficient resources 6.23G 6.81G
8m 2c9g load failed 7.22G

elstic avatar Apr 26 '23 01:04 elstic

@elstic so shall we close this issue now?

yanliang567 avatar Apr 27 '23 01:04 yanliang567

@elstic so shall we close this issue now?

Yes, I'll close this issue

elstic avatar Apr 27 '23 01:04 elstic