connect-es realization

[githublaohu]

What is the purpose of the change

实现从MySQL同步数据到es

To synchronize data from Mysql to es

Brief changelog

实现connect-es整体项目

Realize the whole project of connect es

Verifying this change

集群联调环境在建设中

Cluster joint debugging environment is under construction

Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

• [x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
• [x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
• [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
• [x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
• [x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test to make sure integration-test pass.
• [ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

[githublaohu]

1. 配置读取，解析，分析待完成
2. 集群测试待完成
3. 单元测试待完成
4. 一些细节点没有时间，或者有待讨论

[duhenglucky]

最好是将 ES 的connector 独立出来，其并不依赖上游数据源，上游数据源可能是MySQL，也有可能是日志

[githublaohu]

ok，没问题。

最好是将 ES 的connector 独立出来，其并不依赖上游数据源，上游数据源可能是MySQL，也有可能是日志

[vongosling]

Glad to see this issue re-pickup and look forward to further optimization of CDC. I'm happy to follow up on this issue, Fingers crossed!

[vongosling]

any process?

[githublaohu]

Development, unit test and integration test have been completed. In addition to MySQL, it can be compatible with other simple data sources. The upstream MySQL source (rocketmq connect mysql and rocketmq MySQL) has some problems，so it is difficult to test the integration

[vongosling]

I see, let's look for help from the community.

[githublaohu]

yes. The upstream source cannot provide the downstream data completely. Can the existing source and sink be combined to complete a business process. Connect needs an overall driver. Thank you.