milvus icon indicating copy to clipboard operation
milvus copied to clipboard

enhance: Enable dynamic update loaded collection's replica

Open weiliu1031 opened this issue 1 year ago • 2 comments

issue: #35821 After collection loaded, if we need to increase/decrease collection's replica, we need to release and load it again.

milvus offers 4 solution to update loaded collection's replica, this PR aims to dynamic change the replica number without release, and after replica number changed, milvus will execute load replica or release replica in async, and the replica loaded status can be checked by getReplicas API.

Notice that if set too much replicas than querynode can afford,the new replica won't be loaded successfully until enough querynode joins.

weiliu1031 avatar Aug 29 '24 08:08 weiliu1031

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Aug 29 '24 08:08 mergify[bot]

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Aug 29 '24 09:08 mergify[bot]

rerun ut

weiliu1031 avatar Aug 30 '24 07:08 weiliu1031

rerun ut

wayblink avatar Aug 30 '24 10:08 wayblink

Codecov Report

Attention: Patch coverage is 75.37797% with 114 lines in your changes missing coverage. Please review.

Project coverage is 82.32%. Comparing base (c50fe71) to head (83ee29d). Report is 17 commits behind head on master.

Files with missing lines Patch % Lines
internal/querycoordv2/job/job_update.go 48.14% 32 Missing and 10 partials :warning:
internal/querycoordv2/server.go 72.09% 8 Missing and 4 partials :warning:
internal/metastore/kv/querycoord/kv_catalog.go 31.25% 10 Missing and 1 partial :warning:
internal/querycoordv2/services.go 88.42% 8 Missing and 3 partials :warning:
internal/querycoordv2/meta/resource_manager.go 0.00% 9 Missing :warning:
internal/rootcoord/alter_database_task.go 73.33% 5 Missing and 3 partials :warning:
internal/querycoordv2/job/job_load.go 72.22% 4 Missing and 1 partial :warning:
internal/querycoordv2/task/executor.go 61.53% 4 Missing and 1 partial :warning:
internal/querycoordv2/meta/replica_manager.go 86.20% 2 Missing and 2 partials :warning:
internal/rootcoord/alter_collection_task.go 88.88% 2 Missing and 1 partial :warning:
... and 2 more
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #35822      +/-   ##
==========================================
+ Coverage   73.38%   82.32%   +8.94%     
==========================================
  Files        1277     1279       +2     
  Lines      150759   151253     +494     
==========================================
+ Hits       110629   124520   +13891     
+ Misses      35137    21717   -13420     
- Partials     4993     5016      +23     
Files with missing lines Coverage Δ
internal/distributed/querycoord/client/client.go 97.53% <100.00%> (+0.06%) :arrow_up:
internal/distributed/querycoord/service.go 77.92% <100.00%> (+0.19%) :arrow_up:
internal/metastore/catalog.go 100.00% <ø> (ø)
internal/querycoordv2/checkers/channel_checker.go 85.61% <100.00%> (+0.19%) :arrow_up:
internal/querycoordv2/checkers/segment_checker.go 90.87% <100.00%> (+1.98%) :arrow_up:
internal/querycoordv2/utils/meta.go 73.10% <100.00%> (+10.28%) :arrow_up:
pkg/config/etcd_source.go 85.71% <100.00%> (ø)
tests/integration/minicluster_v2.go 71.11% <100.00%> (+0.27%) :arrow_up:
internal/querycoordv2/meta/collection_manager.go 88.09% <86.66%> (-0.06%) :arrow_down:
internal/util/mock/grpc_querycoord_client.go 94.73% <0.00%> (-2.57%) :arrow_down:
... and 10 more

... and 263 files with indirect coverage changes

codecov[bot] avatar Sep 03 '24 12:09 codecov[bot]

rerun ut

weiliu1031 avatar Sep 05 '24 09:09 weiliu1031

rerun ut

weiliu1031 avatar Sep 06 '24 14:09 weiliu1031

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Sep 09 '24 10:09 mergify[bot]

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Sep 20 '24 13:09 mergify[bot]

@weiliu1031 go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify[bot] avatar Sep 20 '24 13:09 mergify[bot]

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Sep 23 '24 03:09 mergify[bot]

/hold

weiliu1031 avatar Sep 23 '24 11:09 weiliu1031

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify[bot] avatar Sep 23 '24 13:09 mergify[bot]

@weiliu1031 go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify[bot] avatar Sep 23 '24 13:09 mergify[bot]

/run-cpu-e2e

weiliu1031 avatar Sep 24 '24 02:09 weiliu1031

/unhold

weiliu1031 avatar Sep 24 '24 08:09 weiliu1031

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: congqixia, weiliu1031

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

sre-ci-robot avatar Sep 25 '24 02:09 sre-ci-robot