tiflow icon indicating copy to clipboard operation
tiflow copied to clipboard

schema(ticdc): use node level schema storage

Open CharlesCheung96 opened this issue 1 month ago • 6 comments

What problem does this PR solve?

Issue Number: close #xxx

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)

Scenario: incremental scan Deploy ticdc cluster(7 nodes, 8c16g, replicating 150k tables, 1 changefeed) with the following configuration:

[debug.scheduler]
add-table-batch-size = 1000
  1. Before this pr: The maximum memory of the owner node is 15.6GB https://snapshots.raintank.io/dashboard/snapshot/LbadunzKLNNUuKz9jN5xwWfdFdfK8Uph image

  2. This pr: The maximum memory of the owner node is 11GB https://snapshots.raintank.io/dashboard/snapshot/siOmwwftr3h7jYaoruECv7hb8Vl4bHwZ image

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note

`Optimize the memory usage of the ticdc owner node in the scenario of synchronizing a large number of tables`.

CharlesCheung96 avatar May 28 '24 02:05 CharlesCheung96