paimon icon indicating copy to clipboard operation
paimon copied to clipboard

[flink] Delete in flink should produce changelog no matter what

Open leaves12138 opened this issue 2 years ago • 1 comments

Purpose

Delete one partition in flink will not produce changelog. But anyway, it should. The pr has done two things:

1、Before delete push down, check the row count, if row number is bigger than 100_0000, we generate a job to finish delete. (Do not apply push down)\

2、Delete the partition by per-record set (set the record kind to RowKing.DELETE, then write it back) in Delete push down action.

Tests

API and Format

Documentation

leaves12138 avatar Dec 28 '23 09:12 leaves12138

Can you also optimize Spark?

Done

leaves12138 avatar Apr 24 '24 09:04 leaves12138

+1

JingsongLi avatar May 04 '24 12:05 JingsongLi