[Bug] full-compaction.delta-commits

[junmuz]

Search before asking

• [x] I searched in the issues and found nothing similar.

Paimon version

77f130bde03381e3b26b86e4f0e95cff5c1dc622

Compute Engine

Flink

Minimal reproduce step

I created a pkey Paimon table with the following table properties with Iceberg compatibility enabled.

changelog-producer: lookup
full-compaction.delta-commits: 1
metadata.iceberg.storage-location: table-location
metadata.iceberg.manifest-legacy-version: true
metadata.iceberg.manifest-compression: snappy
metadata.iceberg.previous-versions-max: 24
metadata.iceberg.storage: hive-catalog
metadata.iceberg.hive-client-class: com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient
metadata.iceberg.glue.skip-archive: true
metadata.iceberg.uri: ""
metadata.iceberg.hive-conf-dir: ""
metadata.iceberg.hadoop-conf-dir: ""

I performed the inserts and update operations on the above created table.

What doesn't meet your expectations?

After performing the updates, I don't see the updates getting reflected from Iceberg compatibility layer. I was following the source code, and it appears that the delta-commits trigger only happens when the changelog-producer is set to full-compaction. The delta-commits is handled only inside the GlobalFullCompactionSinkWrite, which only gets invoked when the changelog-producer is full-compaction. Inside the docs as well, the full-compaction.delta-commits is only mentioned for Full Compaction changelog-producer.

When we have changelog-producer as lookup, it relies on the LookupSinkWrite that doesn't take care of the delta-commit property.

Anything else?

Based on the above behavior, I think of the following 2 solutions.

1. Either allow the full-compaction.delta-commits for other changelog producers as well. This will have a drawback increasing the cost of commit operations as full-compaction will be triggered everytime.
2. Improve the documentation, and not allow setting full-compaction.delta-commits property when changelog producer is different from full-compaction.

Let me know about your thoughts.

Are you willing to submit a PR?

• [ ] I'm willing to submit a PR!

[ArnavBalyan]

Hi thanks for raising this @junmuz, I'll raise a patch for #2, can probably fix #1 long term