flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

[FLINK-35243][cdc] Supports more schema change event types & backfill pre-schema payload for sink

Open yuxiqian opened this issue 1 year ago • 3 comments

This PR closes FLINK-35243:

  • Supports AlterColumnCommentEvent, AlterTableCommentEvent, DropTableEvent, RenameTableEvent, and TruncateTableEvent in pipeline framework
  • Adds SchemaChangeEventWithPreSchema interface, allowing SchemaRegistry backfilling schema before change payload
  • Fixed MySQL pipeline source doesn't capture ALTER TABLE ... MODIFY COLUMN DDL
  • Adds corresponding test cases

This requires #3339 being merged first.

yuxiqian avatar May 06 '24 10:05 yuxiqian

@PatrickRen @ruanhang1993 PTAL

yuxiqian avatar May 08 '24 10:05 yuxiqian

DropTableEvent TruncateTableEvent can be closed. Sometimes, tables cannot be deleted synchronously.

melin avatar May 10 '24 05:05 melin

DropTableEvent TruncateTableEvent can be closed. Sometimes, tables cannot be deleted synchronously.

Hi @melin, thanks for your comments. It's a common use case to apply some "safe" schema change events (like AddColumnEvent) and ignore "dangerous" events (like Truncate / Drop tables). So we're designing a new feature allowing users to configure schema change behaviour for each type of schema change events in FLINK-35242. Also, we're trying to expose interfaces allowing Sink connectors to report their supported kind of schema change events.

This PR is still in early draft stage, so feel free to leave your comments if you have any other concerns.

yuxiqian avatar May 10 '24 06:05 yuxiqian

support add/drop primary key event

melin avatar Aug 02 '24 05:08 melin

Thanks for @melin's suggestion, will be working on this after #3339 got merged.

yuxiqian avatar Aug 02 '24 05:08 yuxiqian

Seems it's not meaningful to rebase, since there are too many conflicts. Will open another PR to address this.

yuxiqian avatar Aug 08 '24 08:08 yuxiqian