[Feature] Create Dela log after each commit

[jles01]

Search before asking

• [X] I searched in the issues and found nothing similar.

Motivation

Motivation is to make paimon readable by databricks sql endpoints

Solution

After each commit we can create delta log which will have a list of underlying parquet files

Anything else?

Is there a way to determine parquet files when creating manifest in paimon ?

Are you willing to submit a PR?

• [X] I'm willing to submit a PR!

[jles01]

Can anyone help me. I want to create delta lag after each paimon commit (also with delta delete vectors etc) question how can I retrieve the list of files created by paimon.

[JingsongLi]

This is a big feature, it is not easy to do, but we will consider this.

[jles01]

Hey I have a working prototype and will create WIP PR. Already tested against databricks sql warehouse and seems to work. Just need to add more coverage tests