[Feature][Connector] New SeaTunnel API Connectors

[Hisoka-X]

trafficstars

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Search before asking

• [X] I had searched in the issues and found no similar issues.

Describe the proposal

Our community design a new connector api for connector, so we don't write same code in flink and spark twice. The progress are great, we can run example at now. But also have many work to do.

Task list

Core

• [ ] The new connector test with differcult environment.
• [X] Add the new connector starter.
• [X] Add new seatunnel connector API doc.
• [X] Add new seatunnel connector example.
• [ ] Split the translation in distribution.
• [ ] Add new connector e2e test.
• [ ] ....
• [x] Merge the api-draft into dev.

Connector

The guide of new connector contribute check

No	Connector	priority	difficulty	Source/Sink	Contributer	Status	Issue/PR
1	Console	high	low	Sink	@ruanwenjun	Done	#1864
2	Fake	high	low	Source	@ruanwenjun	Done	#1864
3	Doris	high	high	Source	@2013650523		https://github.com/apache/incubator-seatunnel/pull/2536
4	Doris	high	high	Sink	@hk-lrzy		https://github.com/apache/incubator-seatunnel/pull/2586
5	Druid	high	middle	Source	@guanboo	Done	https://github.com/apache/incubator-seatunnel/pull/2651
6	Druid	high	middle	Sink	@guanboo	Done	https://github.com/apache/incubator-seatunnel/pull/2651
7	ElasticSearch	high	high	Source	@iture123		https://github.com/apache/incubator-seatunnel/pull/2821
8	ElasticSearch	high	high	Sink	@iture123	Done	#2330
9	ClickHouse	high	high	Source	@Hisoka-X	Done	#2051
10	ClickHouse	high	high	Sink	@Hisoka-X	Done	#2051
11	JDBC	high	high	Source	@ic4y	Done	#2048
12	JDBC	high	high	Sink	@ic4y	Done	#2048
13	FeiShu	low	low	Source	@TyrantLucifer
14	FeiShu	low	low	Sink	@TyrantLucifer	Done
15	File(Local)	high	middle	Source	@TyrantLucifer	Done
16	File(Local)	high	middle	Sink	@EricJoy2048	Done	#2117
17	File(S3)	high	middle	Source	@TyrantLucifer
18	File(S3)	high	middle	Sink	@TyrantLucifer
19	File(HDFS)	high	high	Source	@TyrantLucifer	Done
20	File(HDFS)	high	high	Sink	@EricJoy2048	Done
21	File(OSS)	high	middle	Source	@TyrantLucifer	Done
22	File(OSS)	high	middle	Sink	@TyrantLucifer	Done
23	Hudi	high	high	Source	@Emor-nj	Done	#2147
24	Hudi	high	high	Sink	@Emor-nj
25	Icebreg	high	middle	Source	@hailin0	Done	#2615
26	Icebreg	high	middle	Sink	@2013650523
27	Kafka	high	middle	Source	@Hisoka-X	Done
28	Kafka	high	middle	Sink	@ruanwenjun	Done
29	Kudu	high	high	Source	@2013650523	Done	#2254
30	Kudu	high	high	Sink	@2013650523	Done	#2254
31	MongoDB	high	middle	Source	@wuchunfu	Done
32	MongoDB	high	middle	Sink	@wuchunfu	Done
33	Neo4j	middle	middle	Source	@getChan		https://github.com/apache/incubator-seatunnel/pull/2777
34	Neo4j	middle	middle	Sink	@getChan	Done	#2434
35	Phoenix	middle	middle	Source	@531651225	Done	#2499
36	Phoenix	middle	middle	Sink	@531651225	Done	#2499
37	Redis	high	middle	Source	@TyrantLucifer	Done
38	Redis	high	middle	Sink	@TyrantLucifer	Done
39	Socket	high	low	Source	@zhuangchong	Done	#1999
40	Socket	high	low	Sink	@531651225	Done	#2549
41	[JDBC]Tidb	high	low	Source	@xbkaishui
42	[JDBC]Tidb	high	low	Sink	@xbkaishui
43	Webhook	high	middle	Sink	@TyrantLucifer	Done	#2348
44	InfluxDB	middle	middle	Source	@531651225		https://github.com/apache/incubator-seatunnel/pull/2697
45	InfluxDB	middle	middle	Sink	@Jellal-HT
46	Pulsar	high	middle	Source	@ashulin	Done	#1984
47	Pulsar	high	middle	Sink	@FlechazoW
48	Email	middle	low	Sink	@2013650523	Done	#2304
49	Assert	low	low	Sink	@lhyundeadsoul	Done
50	Redshift	high	high	Source
51	Facebook Marketing	middle	middle	Source
52	HubSpot	middle	middle	Source
53	Instagram	middle	middle	Source
54	Bing ADs	middle	middle	Source
55	Google Analytics	middle	middle	Source	@MRYOG
56	Intercom	middle	middle	Source
57	Zendesk	middle	middle	Source
58	TikTok Marketing	middle	middle	Source
59	Salesforce	middle	middle	Source
60	LinkedIn Ads	middle	middle	Source
61	Stripe	middle	middle	Source
62	Sentry	middle	middle	Sink	@Saintyang	done	#2244
63	DingTalk	low	low	Source	@MRYOG		https://github.com/apache/incubator-seatunnel/pull/2757
63	DingTalk	low	low	Sink	@MRYOG	Done	#2257
64	IoTDB	high	high	Source	@CalvinKirs	Done
65	IoTDB	high	high	Sink	@hailin0	Done
66	TD-engine	middle	middle	Source	@lhyundeadsoul		https://github.com/apache/incubator-seatunnel/pull/2707
67	TD-engine	middle	middle	Sink	@lhyundeadsoul
68	HBase	high	high	Source	@zhuzhengjun01
69	HBase	high	high	Sink	@zhuzhengjun01
70	Hive	high	high	Source	@CalvinKirs	Done
71	Hive	high	high	Sink	@EricJoy2048	Done
72	HTTP	high	middle	Source	@zhuangchong	Done	#2012
72	StarRocks	high	high	Sink	@tonyDong-code
73	StarRocks	high	high	Source	@wangw9420
74	ADB PostgreSQL	high	low	Sink	@etcZYP
75	Greenplum	high	high	Source	@hailin0	Done	#2429
76	Greenplum	high	high	Sink	@hailin0	Done	#2429
77	OceanBase	middle	high	Sink	@silenceland
78	OceanBase	middle	high	Source	@silenceland
79	DB2	high	high	Sink	@laglangyue		#2410
80	DB2	high	high	Source	@laglangyue		#2410
81	BigSource	middle	middle	Source
82	Github	low	low	Source	@MonsterChenzhuo
83	Enterprise WeChat	low	low	Sink	@531651225	Done	#2412
84	Slack	middle	low	Sink	@Charlie17Li
85	Databricks Lakehouse	high	high	Sink
86	Snowflake	middle	high	Sink	@Tangruilin
87	Snowflake	middle	high	Source	@Tangruilin
88	[Jdbc]Sql-Server	middle	low	Sink	@liugddx		https://github.com/apache/incubator-seatunnel/pull/2646
89	[Jdbc]Sql-Server	middle	low	Source	@liugddx		https://github.com/apache/incubator-seatunnel/pull/2646
90	[Jdbc]Oracle	high	low	Sink	@liugddx		https://github.com/apache/incubator-seatunnel/pull/2550
91	[Jdbc]Oracle	high	low	Source	@liugddx		https://github.com/apache/incubator-seatunnel/pull/2550
92	[JDBC]Rds	high	low	Sink	@s7monk		https://github.com/apache/incubator-seatunnel/issues/2829
93	[JDBC]Rds	high	low	Source	@s7monk		https://github.com/apache/incubator-seatunnel/issues/2829
94	[JDBC]SqlLite	middle	low	Sink	@maruko-code
95	[JDBC]SqlLite	middle	low	Source	@Caribbeanz
96	[JDBC]DM(达梦)	middle	low	Source	@laglangyue	done	#2377
97	[JDBC]DM(达梦)	middle	low	Sink	@laglangyue	done	#2377
98	Cassandra	middle	middle	Sink	@bigdataf
99	Cassandra	middle	middle	Source	@bigdataf
100	[File]excel	middle	low	Sink	@Bingz2		https://github.com/apache/incubator-seatunnel/pull/2585
101	[File]excel	middle	low	Source	@MonsterChenzhuo
102	[File]JSON	low	low	Sink	@hailin0	Done
103	[File]JSON	low	low	Source	@TyrantLucifer	Done
104	MaxCompute	middle	middle	Source	@longer-jl
105	MaxCompute	middle	middle	Sink	@longer-jl
106	TDSql	middle	middle	Source	@dzzxjl
107	OpenMLDB	middle	high	Source	@TyrantLucifer
107	OpenMLDB	middle	high	Sink	@Dlimeng
108	Ftp	middle	middle	Sink	@chessplay	done	https://github.com/apache/incubator-seatunnel/pull/2774
109	Ftp	middle	middle	Source	@guanboo		https://github.com/apache/incubator-seatunnel/pull/2774
110	GaussDB	middle	middle	Source	@Builder34
111	GaussDB	middle	middle	Sink	@Builder34
110	Teradata	middle	middle	Source
111	Teradata	middle	middle	Sink
112	SFTP	middle	middle	Sink	@TyrantLucifer
113	SFTP	middle	middle	Source	@TyrantLucifer
114	DataHub	middle	middle	Sink	@chessplay	done
115	SAP HANA	middle	middle	Source
116	SAP HANA	middle	middle	Sink
117	Flink Table Store	high	high	Sink	@iture123
118	Flink Table Store	high	high	Source	@zhaomin1423
119	Sentry	middle	middle	Source
120	Vertica	middle	middle	Source
121	Vertica	middle	middle	Sink
122	Kylin	middle	middle	Source	@531651225
123	Kylin	middle	middle	Sink
124	Neocrm	middle	middle	Source
125	TiDB	middle	middle	Source	@Xuxiaotuan		https://github.com/apache/incubator-seatunnel/pull/2830
126	TiDB	middle	middle	Sink	@Xuxiaotuan		https://github.com/apache/incubator-seatunnel/pull/2830

[EricJoy2048]

Great job! I can work for Hive Sink Connector.

[ashulin]

I can work for Apache pulsar source connector.

[ic4y]

Great. Can you assign me to MongoDB connector.

[qigengwei]

Good job!

[ic4y]

Sorry i plan to do JDBC Sink first. Please reassign JDBC Sink to me

[wolfboys]

Hbase, Redis thx.

[EricJoy2048]

Since most of the work in hive's sink connector is the writing of HDFS files, which is very similar to the work in file sink, I also complete the file sink connector.

[zhuangchong]

socket source, thx.

#1999

[zhuangchong]

Http source thx.

[ic4y]

jdbcSink is done, please assign jdbcSource to me. Thanks

[legendtkl]

Socket sink

[wuchunfu]

MongoDB thx

[davidzollo]

good job, has assigned to you

MongoDB thx

[hk-lrzy]

i can work for doris sink first. can you assign it to me?

[iture123]

I can work for ElasticSearch Source and Sink Connector.

it is too dependent now,because including transport client、RestHighLevelClient and RestLowLevelClient in Seatunnel.We can use RestLowLevelClient to support all versions of ElastichSearch.

[2013650523]

Kudu thx.

[Emor-nj]

I can work for hudi Source and Sink Connector. thx

[EricJoy2048]

@Hisoka-X Can we split file sink into HdfsFileSink, LocalFileSink and S3FileSink or other? I implemented HdfsFileSink and LocalFileSink, and I hope some people can implement S3FileSink.

[Hisoka-X]

@Hisoka-X Can we split file sink into HdfsFileSink, LocalFileSink and S3FileSink or other? I implemented HdfsFileSink and LocalFileSink, and I hope some people can implement S3FileSink.

Got it

[lhyundeadsoul]

I can work for tidb Source and Sink Connector. thx

Can you add Assert sink ? @Hisoka-X thx

[Hisoka-X]

I can work for tidb Source and Sink Connector. thx

Can you add Assert sink ? @Hisoka-X thx

Got it

[2013650523]

I can work for s3 Sink Connector. thx

[CalvinKirs]

I can work for s3 Sink Connector. thx

welcome:) assigned to you.

[ovasty]

Need Apache IoTDB, thanks

[FlechazoW]

I can work for pulsar sink connector. thx

[FlechazoW]

How about add TD-engine?

[EricJoy2048]

How about add TD-engine?

I think it's a good idea.

[EricJoy2048]

Need Apache IoTDB, thanks

cc @Hisoka-X

[EricJoy2048]

I can work for pulsar sink connector. thx

You can welcome. cc @Hisoka-X

[lhyundeadsoul]

@EricJoy2048 @Hisoka-X @CalvinKirs I'm already in touch with TDengine guys. So please give it to me.