inlong icon indicating copy to clipboard operation
inlong copied to clipboard

[INLONG-5460][Sort][Manager] Support Apache Hudi

Open Jellal-HT opened this issue 2 years ago • 3 comments

Prepare a Pull Request

  • Fixes #5099

Motivation

Support Apache Hudi in sort module and manager module

Modifications

  • Extend a new Extract Node for Apache Hudi
  • Extend a new Load Node for Apache Hudi
  • Add the corresponding flink connectors for Apache Hudi
  • Extend Extract Node and Load Node in manager module for apache Hudi

Verifying this change

  • [x] This change added tests and can be verified as follows:
    • add the Unit test of HudiLoadNode HudiLoadNodeTest
    • add the Unit test of HudiExtractNode HudiExtractNodeTest
    • add Hudi source service test HudiSourceServiceTest for manager module
    • add Hudi sink service test HudiSinkServiceTest for manager module

Jellal-HT avatar Aug 15 '22 07:08 Jellal-HT

@Jellal-HT great job. please add the license file for the new dependency, and you can refer to https://inlong.apache.org/zh-CN/community/how-to-maintain-3rd-party-dependencies/.

otherwise, the workflow runs fail. Please check again.

dockerzhang avatar Aug 15 '22 09:08 dockerzhang

hi @Jellal-HT A small suggested version upgrade to 0.12 for hudi.

XuQianJin-Stars avatar Aug 22 '22 07:08 XuQianJin-Stars

Are you missing the step of bundling extract node and load node into the InLong sort ?

duowan1520 avatar Aug 24 '22 11:08 duowan1520

duplicated with https://github.com/apache/inlong/pull/6789 and https://github.com/apache/inlong/pull/6790, close it.

dockerzhang avatar Dec 09 '22 03:12 dockerzhang