inlong icon indicating copy to clipboard operation
inlong copied to clipboard
verified publisher icon apache

[Umbrella] Add sort connectors which support Flink 1.15

Open EMsnap opened this issue 2 years ago • 5 comments

Describe the proposal

Currently there are two sort connectors that can be running on both flink1.13 and flink1.15 which are starrocks and PostGreSQL

We should provide flink 1.15 support on all connectors that can only run on flink 1.13

They are :

  1. mysql-cdc
  2. elasticSearch
  3. hdfs
  4. iceberg
  5. jdbc
  6. kafka
  7. pulsar
  8. tubemq
  9. mongoDb-cdc
  10. sqlServer-cdc
  11. redis
  12. doris
  13. hbase
  14. hudi
  15. kudu
  16. oracle-cdc
  17. hive

How to tackle down

Take hudi connector as an Example

  1. Please make sure that you have read and understand this post : https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sourcessinks/
  2. Please make sure that you have read and understand every line of code in hudi connector on flink 1.13 (If things are not clear to you please present a issue or discussion on https://github.com/apache/inlong)
  3. Most of the systems have their flink connectors on their github page, for instance hudi (https://github.com/apache/hudi/blob/master/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java), we will import this dependency and add our inlong-specific features (metric, audit) on them.
  4. Make sure you add license after modify the TableFactory in /licenses/inlong-sort-connectors
  5. Run some tests on flink1.15 and make sure your connector works
  6. Modify inlong-distribution and make sure the connectors will be compiled just fine

Make reference for these two connectors: https://github.com/apache/inlong/pull/8235 https://github.com/apache/inlong/pull/8193

Task list

  • [x] https://github.com/apache/inlong/issues/8643
  • [x] https://github.com/apache/inlong/issues/8358
  • [x] https://github.com/apache/inlong/issues/8695
  • [x] https://github.com/apache/inlong/issues/8999
  • [x] https://github.com/apache/inlong/issues/9009
  • [x] https://github.com/apache/inlong/issues/8967
  • [x] https://github.com/apache/inlong/issues/8922
  • [x] https://github.com/apache/inlong/issues/8065
  • [x] https://github.com/apache/inlong/issues/8062
  • [x] https://github.com/apache/inlong/issues/8994
  • [x] https://github.com/apache/inlong/issues/8959
  • [ ] https://github.com/apache/inlong/issues/9010
  • [x] https://github.com/apache/inlong/issues/8948
  • [x] https://github.com/apache/inlong/issues/8464

InLong Component

InLong Sort

Are you willing to submit PR?

  • [ ] Yes, I am willing to submit a PR!

Code of Conduct

EMsnap avatar Jun 27 '23 08:06 EMsnap

hi， @EMsnap i want add kafka connector on flink 1.15. The issue I submitted is https://github.com/apache/inlong/issues/8358

hnrainll avatar Jun 28 '23 15:06 hnrainll

@hnrainll thanks, #8358 was assigned to you.

dockerzhang avatar Jun 29 '23 02:06 dockerzhang

I want add TubeMQ connector on flink 1.15. The issue I submitted is https://github.com/apache/inlong/issues/8922

fancycoderzf avatar Sep 17 '23 20:09 fancycoderzf

This issue is stale because it has been open for 60 days with no activity.

github-actions[bot] avatar Feb 27 '24 01:02 github-actions[bot]

This issue is stale because it has been open for 60 days with no activity.

github-actions[bot] avatar Aug 26 '24 01:08 github-actions[bot]