paimon
paimon copied to clipboard
[spark] add spark v2 write
Purpose
Linked issue: part of https://github.com/apache/paimon/issues/4816
Support spark datasource v2 write path, reduce write serialization overhead and accelerate the process of writing to primary key tables in Spark. Currently only added support for fixed-bucket and bucket unware table, unsupported table types would fallback to v1 write.
Tests
V2WriteRequireDistributionTest SparkV2WriteITCase
API and Format
No
Documentation
Added a Spark SQL propertie spark.sql.paimon.use-v2-write