Eventual-Inc
Support Paimon lake format
Is your feature request related to a problem?
Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations. Paimon innovatively combines lake format and LSM structure, bringing realtime streaming updates into the lake architecture. Daft only supports table format including Iceberg, Hudi etc at present.
Describe the solution you'd like
Support Paimon lake format, which integration could follow the pattern of Iceberg.
Describe alternatives you've considered
No response
Additional Context
No response
Would you like to implement a fix?
No
Hi @SteNicholas, we currently don't have any plans to support Paimon anytime soon, but would love to hear more about it and why you'd like to use it. If anyone is interested in adding support for Paimon, we're also happy to provide support.
@srilman, Paimon is the most used lake format in Chinese companies. Paimon support brings great benefits to the Daft practice of Chinese companies. BTW, the Paimon support has already been discussed with @jaychia in community meeting.
We're happy to provide support and discuss integrations. It should hopefully not be too difficult, but would require a Python API for Paimon - is that something that is available?
@jaychia, paimon has python and rust sdk to invoke for support of daft.