flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

[FLINK-35242] Supports per-SE type configuration & tolerance evolution behavior

Open yuxiqian opened this issue 1 year ago • 2 comments

This closes FLINK-35242.

  • Adds TRY_EVOLVE behavior that tolerates exception during metadata applying process
  • Supports configuring various behavior for each schema evolution type with include.schema.changes and exclude.schema.changes sink option
  • Adds schema evolution IT cases

With this change, Schema Operator now stores both "upstream" schema (which always keep up with the structure coming from pipeline source after the transformation) and "evolved" schema (the schema that actually applied to sink). They might differ when:

  • Schema change events are partially or fully ignored (by fine-grained options / IGNORE behavior)
  • Some schema change events failed to apply but pipeline keeps executing (in TRY_EVOLVE mode)

If these cases occur, SchemaOperator needs to cast upstream schema to match the actual evolved version (by adding / removing columns, converting types, etc.) with tolerance (fill in null if meaningful casts are not possible.)

yuxiqian avatar May 21 '24 09:05 yuxiqian

@PatrickRen @leonardBang PTAL

yuxiqian avatar May 24 '24 08:05 yuxiqian

Just added LENIENT schema evolve behavior & rebased to master. Could @loserwang1024 please take a look at this?

Since this PR has grown too big, hopefully this could be reviewed & merged soon to avoid any conflicts hassle in the future. cc @leonardBang

yuxiqian avatar Jun 26 '24 08:06 yuxiqian

@yuxiqian Thanks for the great work, could you rebase to latest master?

leonardBang avatar Jul 12 '24 09:07 leonardBang

Done, rebased with master branch

yuxiqian avatar Jul 12 '24 10:07 yuxiqian

Squashed & Rebased with master.

Since this also changes SchemaManager serialization format, could @lvyanquan please take a look? (Serialization version isn't updated since it's expected that no version will be released between this PR and FLINK-34638.

yuxiqian avatar Aug 07 '24 08:08 yuxiqian