paimon
paimon copied to clipboard
apache
Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
### Purpose Add `--metadata_column` support to Paimon Kafka CDC connector, similar to the already existing options added for MySQL and Postgres: https://github.com/apache/paimon/pull/2077 Supported metadata columns are those on `org.apache.kafka.clients.consumer.ConsumerRecord` i.e.:...
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation flink supported compact_database. https://paimon.apache.org/docs/master/maintenance/dedicated-compaction/#database-compaction-job ### Solution _No response_ ### Anything else? _No response_...
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation The chain table's delta partition should support the stream read ### Solution _No...
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation Support to merge the serval delta partitions and snapshot into a new snapshot...
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation Support to read/write the chain table ### Solution _No response_ ### Anything else?...
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version paimon 1.1 spark 3.2 ### Compute Engine Spark 、 Hive ### Minimal...
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version 1.2 ### Compute Engine all ### Minimal reproduce step ``` @Test public...
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation When will Paimon support Hive 4.x? Hive has now been updated to 4.1.0,...
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version 1.2.0 ### Compute Engine flink ### Minimal reproduce step org.apache.paimon.mergetree.compact.PartialUpdateMergeFunctionTest#testMultiSequenceFields unit test...
### Purpose Compact field files for data evolution table (Which is not compact table just compact single fields in small files) ### Tests ### API and Format ### Documentation
