seatunnel [Feature][flink-connector-hudi] add connector hudi for flink.

[Feature][flink-connector-hudi] add connector hudi for flink.

Open dijiekstra opened this issue 3 years ago • 7 comments

trafficstars

Purpose of this pull request

add Hudi sink #1830. I tried to finish it in two ways

The first is what I push, creating a new Hudi table and inserting it with an INSERT into statement
The second is to emulate the practice in hud-flink by concatenating multiple datastreams to write data to the HUDI table

The first way was lightweight and we didn't need to change it when Hudi was upgraded. The second way will have a lot of duplicate codes from Hudi - Flink, and some codes we need to be inconsistent with hudi -flink, when the HUDI version changes, we need to modify our code.

So, i chooice the first way.

Check list

[x] Code changed are covered with tests, or it does not need tests for reason:
[x] If any new Jar binary package adding in your PR, please add License Notice according New License Guide
[x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

May 11 '22 15:05 dijiekstra

Hi, the build failed. Please fix it first.

May 16 '22 02:05 Hisoka-X

Hi, the build failed. Please fix it first.

I check it, it seems not my bug. Can help me retrigger ci?

May 16 '22 13:05 dijiekstra

Can it use flink sql to create hudi connector table? Hudi connector SQL table supports more. Synchronization of update and delete operations can be supported.

May 17 '22 07:05 waywtdcc

Can it use flink sql to create hudi connector table? Hudi connector SQL table supports more. Synchronization of update and delete operations can be supported.

if use seatunnel-flink-sql, may be we could not use other seatunnel-flink-connector ?

May 17 '22 13:05 dijiekstra

How is the progress of this PR?@CalvinKirs