milvus icon indicating copy to clipboard operation
milvus copied to clipboard

enhance: resource estimate improvement

Open chyezh opened this issue 9 months ago • 25 comments

issue: #32963

  • Add Exception to file class to avoid exception ignore.

  • Add GetResourceUsage method for index, column and segment.

  • Use bin log size but not mem size in CU of search and query.

  • Use in-use disk size in disk cache weight if segment is loaded.

  • Enable cache metric itself, remove redundant metric at search/query path.

chyezh avatar May 10 '24 16:05 chyezh

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chyezh To complete the pull request process, please assign wxyucs after the PR has been reviewed. You can assign the PR to them by writing /assign @wxyucs in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

sre-ci-robot avatar May 10 '24 16:05 sre-ci-robot

@chyezh ut workflow job failed, comment rerun ut can trigger the job again.

mergify[bot] avatar May 10 '24 17:05 mergify[bot]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar May 10 '24 19:05 mergify[bot]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar May 11 '24 04:05 mergify[bot]

@chyezh ut workflow job failed, comment rerun ut can trigger the job again.

mergify[bot] avatar May 11 '24 04:05 mergify[bot]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar May 11 '24 08:05 mergify[bot]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar May 11 '24 13:05 mergify[bot]

/run-cpu-e2e

chyezh avatar May 11 '24 15:05 chyezh

rerun ut

chyezh avatar May 11 '24 15:05 chyezh

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar May 11 '24 16:05 mergify[bot]

/run-cpu-e2e

chyezh avatar May 11 '24 17:05 chyezh

Codecov Report

Attention: Patch coverage is 70.79646% with 99 lines in your changes are missing coverage. Please review.

Project coverage is 82.08%. Comparing base (8a9a421) to head (7935ad6). Report is 28 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff            @@
##           master   #32964    +/-   ##
========================================
  Coverage   82.07%   82.08%            
========================================
  Files        1007      999     -8     
  Lines      127647   128008   +361     
========================================
+ Hits       104767   105074   +307     
- Misses      18903    18960    +57     
+ Partials     3977     3974     -3     
Files Coverage Δ
internal/core/src/common/Types.h 31.45% <ø> (+3.43%) :arrow_up:
internal/core/src/index/Index.h 100.00% <ø> (ø)
internal/core/src/index/InvertedIndexTantivy.cpp 89.47% <100.00%> (ø)
internal/core/src/index/InvertedIndexTantivy.h 47.36% <ø> (ø)
internal/core/src/index/ScalarIndexSort.cpp 74.63% <100.00%> (+0.21%) :arrow_up:
internal/core/src/index/ScalarIndexSort.h 57.14% <ø> (ø)
internal/core/src/index/StringIndexMarisa.h 100.00% <ø> (ø)
internal/core/src/index/VectorDiskIndex.h 18.18% <ø> (ø)
internal/core/src/index/VectorMemIndex.h 100.00% <ø> (ø)
internal/core/src/segcore/SegmentGrowingImpl.h 81.35% <100.00%> (+0.99%) :arrow_up:
... and 31 more

... and 197 files with indirect coverage changes

codecov[bot] avatar May 11 '24 17:05 codecov[bot]

@chyezh ut workflow job failed, comment rerun ut can trigger the job again.

mergify[bot] avatar May 12 '24 06:05 mergify[bot]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar May 12 '24 08:05 mergify[bot]

rerun ut

chyezh avatar May 12 '24 12:05 chyezh

rerun ut

chyezh avatar May 12 '24 14:05 chyezh

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar May 12 '24 15:05 mergify[bot]

/run-cpu-e2e

chyezh avatar May 12 '24 15:05 chyezh

rerun ut

chyezh avatar May 12 '24 16:05 chyezh

rerun ut

chyezh avatar May 12 '24 16:05 chyezh

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar May 12 '24 19:05 mergify[bot]

/run-cpu-e2e

chyezh avatar May 13 '24 00:05 chyezh

/lgtm

sunby avatar May 13 '24 02:05 sunby

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar May 13 '24 03:05 mergify[bot]

/run-cpu-e2e

chyezh avatar May 13 '24 03:05 chyezh