paimon icon indicating copy to clipboard operation
paimon copied to clipboard

[spark] add spark v2 write

Open zhongyujiang opened this issue 8 months ago • 0 comments

Purpose

Linked issue: part of https://github.com/apache/paimon/issues/4816

Support spark datasource v2 write path, reduce write serialization overhead and accelerate the process of writing to primary key tables in Spark. Currently only added support for fixed-bucket and bucket unware table, unsupported table types would fallback to v1 write.

Tests

V2WriteRequireDistributionTest SparkV2WriteITCase

API and Format

No

Documentation

Added a Spark SQL propertie spark.sql.paimon.use-v2-write

zhongyujiang avatar Apr 24 '25 14:04 zhongyujiang