paimon
paimon copied to clipboard
[flink] Delete in flink should produce changelog no matter what
Purpose
Delete one partition in flink will not produce changelog. But anyway, it should. The pr has done two things:
1、Before delete push down, check the row count, if row number is bigger than 100_0000, we generate a job to finish delete. (Do not apply push down)\
2、Delete the partition by per-record set (set the record kind to RowKing.DELETE, then write it back) in Delete push down action.
Tests
API and Format
Documentation
Can you also optimize Spark?
Done
+1