tiflow icon indicating copy to clipboard operation
tiflow copied to clipboard

config, scheduler(ticdc): add MaxTableCount and AddTableBatchSize to ChangefeedSchedulerConfig

Open CharlesCheung96 opened this issue 8 months ago • 8 comments

What problem does this PR solve?

Issue Number: close #11216

What is changed and how it works?

  1. Move AddTableBatchSize from server to changefeed config.
  2. Add MaxTableCount to limit the max tables for each capture to avoid unbalance.
  3. Auto adjust batchSize in scheduler_basic if AddTableBatchSize=0. And the adjustment range is 50 to 1000.

Check List

Tests

  • Manual test (add detailed scripts or steps below)

  • Before this PR: The schedule has not been completed in 40 minutes. image

  • This PR: The scheduling was completed in 10min image

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Please refer to [Release Notes Language Style Guide](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/release-notes-style-guide.html) to write a quality release note.

If you don't think this PR needs a release note then fill it with `None`.

CharlesCheung96 avatar May 30 '24 02:05 CharlesCheung96