[FLINK-25568][connectors/elasticsearch] Add Elasticsearch 7 Source

[alpreu]

Draft PR for the Elasticsearch 7 source - not final yet

What is the purpose of the change

This PR introduces a new Source connector for Elasticsearch 7.

Brief change log

• implemented Elasticsearch Source based on Source interface
• moved some classes to be shared between source and sink

Verifying this change

This change added tests and can be verified as follows:

• Unit tests added by this PR

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (yes / no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
• The serializers: (yes / no / don't know)
• The runtime per-record code paths (performance sensitive): (yes / no / don't know)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
• The S3 file system connector: (yes / no / don't know)

Documentation

• Does this pull request introduce a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 31108c79f41b48c4dfe2eca5d65da3986375ebf8 (Mon Dec 20 09:36:26 UTC 2021)

Warnings:

• No documentation files were touched! Remember to keep the Flink docs up to date!
• Invalid pull request title: No valid Jira ID provided

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❓ 3. Needs [attention] from.
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[flinkbot]

CI report:

• f072ee4cbc3712e048c543c13c3b7378bd7d58ed Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[alpreu]

@flinkbot run azure

[alpreu]

@flinkbot run azure

[alpreu]

@flinkbot run azure

[grzegorz8]

@alpreu Hey! Could you tell me what is the status of this pull request, please? Thanks!

[alpreu]

Hi @grzegorz8, I'm not actively working on Flink anymore. I think this PR was kept open as we were waiting to move the ES connector to the external repo. Maybe @AHeise @afedulov can confirm?

[grzegorz8]

Hi @grzegorz8, I'm not actively working on Flink anymore. I think this PR was kept open as we were waiting to move the ES connector to the external repo. Maybe @AHeise @afedulov can confirm?

@alpreu Thank you for your prompt response. I have one more question - how would you estimate the progress of the change in comparison to the scope of FLIP-127? At first glance it looks pretty advanced, except the fact that (1) lookup capability and (2) pushdowns proposed are not implemented yet.

[alpreu]

how would you estimate the progress of the change in comparison to the scope of FLIP-127? At first glance it looks pretty advanced, except the fact that (1) lookup capability and (2) pushdowns proposed are not implemented yet.

@grzegorz8 The scope of this PR was to provide a batch source implementation. Lookup is not supported yet and the pushdowns neither. I believe adding pushdown support to the current state might be easy to do. In any case, feel free to pick up where I left off, it would be great to see this make it into Flink :)

[MartijnVisser]

This PR has been closed as the Elasticsearch connector has been moved to https://github.com/apache/flink-connector-elasticsearch - If there's a volunteer who would like to work on this, it's much appreciated :)