elstic
elstic
Deployment method:kuernetes Standalone Deployed v2.2.6 instance, inserted 100,000 128-dimensional data into 85 collections, inserted 10,000 data into 120 collections, then upgraded to v2.2.9; waited 2 days, cpu usage went up...
This issue has been verified and fixed .
case : test_concurrent_locust_100m_ivf_sq8_ddl_dql_filter_replica2_cluster After running the test and waiting a few hours without doing anything, querynode memory rises. server : ``` fouramf-stbsk-99-2262-etcd-0 1/1 Running 0 3m54s 10.104.16.14 4am-node21 fouramf-stbsk-99-2262-etcd-1 1/1...
case: test_concurrent_locust_100m_ivf_sq8_ddl_dql_filter_kafka_cluster image: master-20230905-fb0705df querynode memory up at 28h  querynode memory went from 1.19g to 5.54g server: ``` fouramf-9jq2k-13-8541-etcd-0 1/1 Running 0 2d3h 10.104.17.83 4am-node23 fouramf-9jq2k-13-8541-etcd-1 1/1 Running 0...
#### [cluster] server: ``` fouramf-stable-1682190000-8-etcd-0 1/1 Running 0 6m51s 10.104.16.203 4am-node21 fouramf-stable-1682190000-8-etcd-1 1/1 Running 0 6m50s 10.104.18.238 4am-node25 fouramf-stable-1682190000-8-etcd-2 1/1 Running 0 6m50s 10.104.15.127 4am-node20 fouramf-stable-1682190000-8-milvus-datacoord-99ddfdf47-z7cww 1/1 Running 1 (2m51s...
> @elstic I tried 2.2.0-20230621-bd75f4c3 in standalone mode and it turned that this problem has been fixed. Could you verify and decide whether this issue can be resolved or not?...
> /assign @elstic this has been fixed with #23334 After verification, this issue has been fixed, querynode memory usage balancing. verification version: 2.2.0-20230418-e1122c2a.
> can we change the pod limit to 2G and increase the ingest concurrency and see if it OOMed? > /assign @elstic please retry as suggested above, and did this...
> can we change the pod limit to 2G and increase the ingest concurrency and see if it OOMed? Set limit 2G and no OOM occurs. Just insert deny ....
> /assign @elstic After verification, datanode memory rise still exists . use image: 2.2.0-20230525-ef1a671d argo task : fouramf-4vkjr