paimon icon indicating copy to clipboard operation
paimon copied to clipboard
verified publisher icon apache

[FLINK-30323] Support table statistics in table store

Open FangYongs opened this issue 3 years ago • 4 comments

Support table statistics in table store

FangYongs avatar Dec 07 '22 10:12 FangYongs

Hi @JingsongLi I have rebase from master for this PR, could you help to review again when you get time? THX

FangYongs avatar Jan 18 '23 09:01 FangYongs

I need to reorganize Flink-connector's modules recently, and we can continue to work after that.

JingsongLi avatar Feb 14 '23 11:02 JingsongLi

Hi @JingsongLi I have updated this PR, please help to review it again, thanks

FangYongs avatar Mar 08 '23 06:03 FangYongs

@FangYongs Thanks for the update.

I have thoughts here:

  1. We should avoid copying too much code. I think there is a chance to avoid it
  2. plan is a heavy work, we can avoid it, we can produce splits in compilation phase, and pass these splits to runtime source. Like in https://github.com/apache/flink-table-store/pull/584

JingsongLi avatar Mar 10 '23 10:03 JingsongLi

Hi @JingsongLi I have rebase master for this PR, please help to review it when you're free, thanks

FangYongs avatar Apr 19 '23 03:04 FangYongs

We should avoid copying AbstractFlinkTableFactory. We can use less codes to fix this.

Then maybe we need to create a factory for DataTableSource, it will create DataTableSource for flink-1.14/flink-1.15 and create StatisticsDataTableSource for the later version of flink. What do you think?

FangYongs avatar Apr 28 '23 03:04 FangYongs

We should avoid copying AbstractFlinkTableFactory. We can use less codes to fix this.

Then maybe we need to create a factory for DataTableSource, it will create DataTableSource for flink-1.14/flink-1.15 and create StatisticsDataTableSource for the later version of flink. What do you think?

We can do like https://github.com/apache/incubator-paimon/pull/1010 , this may not look good, but we definitely need to avoid copying large amounts of code, which can be difficult to maintain.

JingsongLi avatar May 06 '23 03:05 JingsongLi

We should avoid copying AbstractFlinkTableFactory. We can use less codes to fix this.

Then maybe we need to create a factory for DataTableSource, it will create DataTableSource for flink-1.14/flink-1.15 and create StatisticsDataTableSource for the later version of flink. What do you think?

We can do like #1010 , this may not look good, but we definitely need to avoid copying large amounts of code, which can be difficult to maintain.

Looks good to me, I have rebased master and updated this PR

FangYongs avatar May 08 '23 09:05 FangYongs