paimon
paimon copied to clipboard
apache
Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
### Search before asking - [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar. ### Motivation It can be big pressure when lots of table use hive locks to...
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version detail: ``` Caused by: com.github.luben.zstd.ZstdException: Data corruption detected at com.github.luben.zstd.ZstdDecompressCtx.decompressByteArray(ZstdDecompressCtx.java:205) at com.github.luben.zstd.Zstd.decompressByteArray(Zstd.java:439)...
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation See more in: https://issues.apache.org/jira/browse/PARQUET-382 This could save lots of resource usage for append...
### Purpose Support Dynamic bloom filter file index.DynamicBloomFilter is largely based of org.apache.hudi.common.bloom.InternalDynamicBloomFilter. Comparing with Hadoop's DynamicBloomFilter, Hoodie's dynamic bloom bounded bloom filter have a bound. Once the entries added...
### Purpose Related to PIP: https://cwiki.apache.org/confluence/display/PAIMON/PIP-37%3A+Introduce+Chain+Table ### Linked issue https://github.com/apache/paimon/issues/6313 ### Tests see: org.apache.paimon.spark.SparkCatalogWithHiveTest#testChainTable ### API and Format Related to PIP: https://cwiki.apache.org/confluence/display/PAIMON/PIP-37%3A+Introduce+Chain+Table ### Documentation Related to PIP: https://cwiki.apache.org/confluence/display/PAIMON/PIP-37%3A+Introduce+Chain+Table
### Purpose Iceberg, lance and object table should not support read and write yet, add wrap for them ### Tests ### API and Format ### Documentation
## Purpose Fixes URI scheme loss issue in blob-as-descriptor mode by replacing `pathlib.Path` with `urlpath.URL` throughout the codebase. This ensures URI schemes (e.g., `oss://`, `s3://`, `file://`) are preserved when handling...
### This PR **Supports combined mode processing when orphan files clean:** - Processes multiple tables within a single DataStream during job graph construction, instead of creating one DataStream per table,...
### Purpose This PR adds support to use Paimon as a Flink CDC Pipeline Source. ### Tests Tests are added to the `org.apache.paimon.flink.pipeline.cdc.source` package of `paimon-flink-cdc` module to verify its...
### Purpose This PR is about to support rowid pushdown for flink, following https://github.com/apache/paimon/pull/6483 Linked issue: None ### Tests Please see the test cases in org.apache.paimon.flink.source.FlinkTableSourceTest ### API and Format...
