paimon
paimon copied to clipboard
[cdc] Optimize SyncDatabaseAction performance by removing listTables calls
Purpose
from #5955
What is the purpose of the change
Optimize SyncDatabaseAction performance by removing expensive listTables operations during initialization, improving scalability for databases with many tables.
Brief change log
- Remove
listTables()call fromRichCdcMultiplexRecordEventParser - Implement lazy table creation in
CdcDynamicTableParsingProcessFunction#processElement - Remove
createdTablesSet to reduce memory usage
Verifying this change
- Verified existing functionality remains intact
Testing
This optimization does not require additional test cases as the existing functionality is already covered by:
SyncDatabaseActionBaseTest.testSyncTablesWithoutDbLists()- validates table filtering logicSyncDatabaseActionBaseTest.testSyncTablesWithDbList()- validates database filtering logicSyncDatabaseActionBaseTest.testSycTablesCrossDB()- validates cross-database filtering scenarios
All these tests create and use RichCdcMultiplexRecordEventParser, ensuring the optimization doesn't break existing functionality.