paimon icon indicating copy to clipboard operation
paimon copied to clipboard
verified publisher icon apache

[spark] Support cross partition write

Open Zouxxyy opened this issue 2 years ago • 3 comments

Purpose

spark support Cross Partitions Upsert Dynamic Bucket Mode

Tests

API and Format

Documentation

Zouxxyy avatar Oct 31 '23 02:10 Zouxxyy

How is this PR going?

waywtdcc avatar Nov 27 '23 02:11 waywtdcc

@waywtdcc cross partition write needs to be refactored, will rebase after refactoring

Zouxxyy avatar Dec 22 '23 07:12 Zouxxyy

Now, How is this PR going?

waywtdcc avatar Mar 21 '24 00:03 waywtdcc

@waywtdcc sorry for delay, we choose to closed it, The current implementation is friendly to streaming scenarios, but the batch cost is too high. We need to find a more efficient implementation in batch scenarios.

Zouxxyy avatar May 31 '24 05:05 Zouxxyy

Doesn't flink batch also support it? The logic of the two should be the same, right?

waywtdcc avatar Jun 14 '24 02:06 waywtdcc