flink-cdc icon indicating copy to clipboard operation
flink-cdc copied to clipboard

[FLINK-36514] Fix unable to override exclude schema types in lenient mode

Open yuxiqian opened this issue 1 year ago • 0 comments

This closes FLINK-36514.

If schema evolution behavior is set to LENIENT, Truncate / Drop table events will be ignored by default. However, there's currently no way for users to override this behavior due to the following code:

// YamlPipelineDefinitionParser#toSinkDef
if (excludedSETypes.isEmpty() && SchemaChangeBehavior.LENIENT.equals(schemaChangeBehavior))
{
    // In lenient mode, we exclude DROP_TABLE and TRUNCATE_TABLE by default. This could be
    // overridden by manually specifying excluded types. 
    Stream.of(SchemaChangeEventType.DROP_TABLE, SchemaChangeEventType.TRUNCATE_TABLE) 
        .map(SchemaChangeEventType::getTag)
        .forEach(excludedSETypes::add);
    }
}

If one wants to exclude no types, it's actually not possible since passing [] is equivalent to passing nothing, and DROP and TRUNCATE events will still be ignored.

This could be fixed by checking if exclude.schema.changes has been set, instead of if parsed excludedSETypes set is empty.

yuxiqian avatar Oct 12 '24 03:10 yuxiqian