paimon icon indicating copy to clipboard operation
paimon copied to clipboard
verified publisher icon apache

[flink] Manage flink version for paimon-flink-common and paimon-flink-cdc

Open FangYongs opened this issue 2 years ago • 8 comments

Purpose

Linked issue: close #2141

Manage flink version

Tests

This feature is covered by existing tests

API and Format

no

Documentation

no

FangYongs avatar Oct 17 '23 05:10 FangYongs

The problem is that flink-kafka-connector will be removed from flink-1.18 repo, so the 1.18-snapshot kafka connector are outdate. We should wait for the flink-1.18 release and then modify the POM.

yuzelin avatar Oct 23 '23 07:10 yuzelin

The problem is that flink-kafka-connector will be removed from flink-1.18 repo, so the 1.18-snapshot kafka connector are outdate. We should wait for the flink-1.18 release and then modify the POM.

Get it. I think we may need to split flink-cdc to sub modules like paimon-flink-cdc-common, paimon-flink-cdc-1.17 and paimon-flink-cdc-1.18. WDYT?

FangYongs avatar Oct 27 '23 07:10 FangYongs

The problem is that flink-kafka-connector will be removed from flink-1.18 repo, so the 1.18-snapshot kafka connector are outdate. We should wait for the flink-1.18 release and then modify the POM.

Get it. I think we may need to split flink-cdc to sub modules like paimon-flink-cdc-common, paimon-flink-cdc-1.17 and paimon-flink-cdc-1.18. WDYT?

No, it is not the good, it is hard to maintain another Flink versions...

JingsongLi avatar Oct 27 '23 07:10 JingsongLi

The problem is that flink-kafka-connector will be removed from flink-1.18 repo, so the 1.18-snapshot kafka connector are outdate. We should wait for the flink-1.18 release and then modify the POM.

Get it. I think we may need to split flink-cdc to sub modules like paimon-flink-cdc-common, paimon-flink-cdc-1.17 and paimon-flink-cdc-1.18. WDYT?

No, it is not the good, it is hard to maintain another Flink versions...

I agree.

yuzelin avatar Oct 27 '23 07:10 yuzelin

@JingsongLi @yuzelin Then how can we maintain cdc for different flink versions? such as flink-1.17 and flink-1.18? If we create paimon-flink-cdc-common, paimon-flink-cdc-1.17 and paimon-flink-cdc-1.17 like paimon-flink-xxx, users can pick the right flink-cdc version as they need

FangYongs avatar Oct 27 '23 07:10 FangYongs

@JingsongLi @yuzelin Then how can we maintain cdc for different flink versions? such as flink-1.17 and flink-1.18? If we create paimon-flink-cdc-common, paimon-flink-cdc-1.17 and paimon-flink-cdc-1.17 like paimon-flink-xxx, users can pick the right flink-cdc version as they need

Actually, the cdc submodule will be shaded to the paimon-flink- jar, and users don't need to download the cdc jar.

yuzelin avatar Oct 27 '23 07:10 yuzelin

FYI, flink-kafka-connector 3.0.1 (for flink 1.18) will be released soon (VOTE: https://lists.apache.org/thread/35gjflv4j2pp2h9oy5syj2vdfpotg486). After that we can update cdc module dependencies.

yuzelin avatar Oct 27 '23 09:10 yuzelin

I think we must maintain only one code in paimon-flink-cdc, even we use some reflection methods.

JingsongLi avatar Oct 27 '23 09:10 JingsongLi