[Feature]Connectors e2e test

[CalvinKirs]

SeaTunnel contains a large number of Connectors, and the basic UT cannot perform system testing in a real environment, so we introduced the seatunnel-e2e module for testing e2e automated testing, we are based on Testcontainers to complete the testing of various Connectors.

About Testcontainers (https://www.testcontainers.org/) Testcontainers is a Java library that supports JUnit tests, providing lightweight, throwaway instances of common databases, Selenium web browsers, or anything else that can run in a Docker container.

If you need to test the Connector based on the Flink engine, you need to inherit org.apache.seatunnel.e2e.flink.FlinkContainer, which is the Flink container we have implemented, which helps you to automatically complete the startup and operation of the Flink engine Execution, container shutdown and other operations, so that you can pay more attention to your own business implementation, if you need to test Kafka, then you need to manually introduce the Docker image of Kafka,

Just like this, see https://www.testcontainers.org/

KafkaContainer kafka = new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:6.2.1"))

If you need to use a custom Docker image, this is also very easy, you can refer to https://www.testcontainers.org/features/creating_images/

For us, the main thing to test is end-to-end data synchronization, you just need to provide a simple configuration file like this, then create the container (maybe not necessary, such as the one below) and start it. Of course, after the execution is completed, there should be a process of data proofreading and verification, which you can complete by yourself.

env {
  # You can set flink configuration here
  execution.parallelism = 1
  job.mode = "BATCH"
  #execution.checkpoint.interval=10000
  #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}

source {
  # This is an example source plugin **only for test and demonstrate the feature source plugin**
    FakeSource {
      result_table_name = "fake"
      field_name = "name,age"
    }

  # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
  # please go to https://seatunnel.apache.org/docs/flink/configuration/source-plugins/Fake
}

transform {
    sql {
      sql = "select name, age from fake"
    }

  # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
  # please go to https://seatunnel.apache.org/docs/flink/configuration/transform-plugins/Sql
}

sink {
  Console {}

  # If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
  # please go to https://seatunnel.apache.org/docs/flink/configuration/sink-plugins/Console
}

Connector	Source/Sink	Contributer	Status
Fake	Source		done
Pulsar	Sink	@cnwyc
Pulsar	Source
Hive	Sink	@TyrantLucifer	Done
Hive	Source	@TyrantLucifer	Done
Hudi	Sink	@bravekong
Kafka	Source	@eyys
Kafka	Sink	@dinggege1024
Jdbc	Source	@zhangyuge1
Jdbc	Sink	@zhangyuge1
HdfsFile	Sink	@TyrantLucifer	Done
LocalFile	Sink	@TyrantLucifer	Done
HdfsFile	Source	@TyrantLucifer	Done
LocalFile	Source	@TyrantLucifer	Done
Socket	Source
ClickHouse	Sink	@zhangyuge1
ClickHouse	Source	@zhangyuge1
Http	Source	@TyrantLucifer
If you are interested in this, please leave a comment and I will assign it to you.

Note: We are only testing seatunnel-connectors-v2

[eyys]

@CalvinKirs Hello, can you assign me the source for Kafka? I want to try it.

[zhangyuge1]

@CalvinKirs Hello, I want to try the source and sink for jdbc. Can you assign them to me?

[dinggege1024]

@CalvinKirs Hello, can you assign me the sink for Kafka? I want to try it.

[TyrantLucifer]

@CalvinKirs Please assign me the sink and the source of Hive,HdfsFile,LocalFile,Http,Socket. I can do these works.

[TyrantLucifer]

@CalvinKirs Please assign me the sink and the source of Hive,HdfsFile,LocalFile,Http,Socket. I can do these works.

BTW, also sink @CalvinKirs

[zhangyuge1]

@CalvinKirs Can you assign me the sink and source of clickhouse? I can do them.

[CalvinKirs]

@CalvinKirs Can you assign me the sink and source of clickhouse? I can do them.

yup, if you have any questions. please call me.

[FlechazoW]

@CalvinKirs Can you assign me the sink and source of pulsar? I can do them.

[FWLamb]

Hello, can you assign me the sink for Clickhouse? My current production environment wants to use new functions. I want to try it.

[TyrantLucifer]

Hello, can you assign me the sink for Clickhouse? My current production environment wants to use new functions. I want to try it.

Sure, good job～

[FWLamb]

I plan to test the hive to clickhouse, but there is no hive related environment for testcontainers. How can I do this？

[bravekong]

@CalvinKirs Can you assign me the sink for Hudi? I'm syncing the data to hudi. I want to try it.

[CalvinKirs]

I plan to test the hive to clickhouse, but there is no hive related environment for testcontainers. How can I do this？

We can use custom images, but the Hive docker images are more troublesome, you could try them.

[cnwyc]

@CalvinKirs Can you assign me the sink for Pulasr? I'm syncing the data to Pulasr. I want to try it.