tiflow
tiflow copied to clipboard
schema(ticdc): use node level schema storage
What problem does this PR solve?
Issue Number: close #xxx
What is changed and how it works?
Check List
Tests
- Unit test
- Integration test
- Manual test (add detailed scripts or steps below)
Scenario: incremental scan Deploy ticdc cluster(7 nodes, 8c16g, replicating 150k tables, 1 changefeed) with the following configuration:
[debug.scheduler]
add-table-batch-size = 1000
-
Before this pr: The maximum memory of the owner node is 15.6GB https://snapshots.raintank.io/dashboard/snapshot/LbadunzKLNNUuKz9jN5xwWfdFdfK8Uph
-
This pr: The maximum memory of the owner node is 11GB https://snapshots.raintank.io/dashboard/snapshot/siOmwwftr3h7jYaoruECv7hb8Vl4bHwZ
Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note
`Optimize the memory usage of the ticdc owner node in the scenario of synchronizing a large number of tables`.