[FLINK-20398][e2e] Migrate test_batch_sql.sh to Java e2e tests framework

[affo]

What is the purpose of the change

Migrate test_batch_sql.sh to end-to-end test frameworks.

Brief change log

• implement BatchSQLTest porting test_batch_sql.sh
• fix issue in getting job ID in FlinkDistribution
• remove test_batch_sql.sh script
• remove test_batch_sql.sh invocations from run-nightly-tests.sh

Verifying this change

This change added tests and can be verified as follows:

• Added integration tests for end-to-end batch mode SQL query execution

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? NA

[flinkbot]

CI report:

• ce5e9b39afd8278d46c3b5b0bf7175e692c232f7 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[affo]

@wuchong @XComp

Hi guys, here is the PR for https://issues.apache.org/jira/browse/FLINK-20398.

I decided to go for LocalStandaloneFlinkResourceFactory as it is already used and part of flink-end-to-end-tests-common other options were:

• MiniCluster
• FlinkContainers (testcontainers-based) Just tell me if you would rather see one of those implementations for some reason 👍

Important concern: This test used to be part of run-nightly-tests.sh, now I think it would run differently 🤔

Should it still run nightly?

[affo]

@XComp thank you for your review, gonna address your feedback today (as I had a week off)

[affo]

@XComp

Required quite of an effort honestly, but here we are with the JUnit5 version of what I had before 👍

This also allowed not to start a separate jar, but to directly include the code in the text and directly run it agains the MiniCluster obtained 👍

Thank you for your detailed review

---

UPDATE

I am still investigating why the test fails in CI as I cannot reproduce that locally... I tried to use java8 for compiling and running, but I hit another error actually 😓

---

UPDATE

rebased and forced pushed, now CI is ok 👍

[affo]

@XComp everything should be ok now 👍

[XComp]

fyi: I will be off for the rest of April and, therefore, wouldn't be able to finalize this PR. You might want to reach out to other committers or expect a delay in my responses.

[affo]

@XComp Glad for your vacation! Finally I also addressed the deprecation warnings and went through the implementation of a custom connector through DynamicTableSource.

It turned out to be quite tough, as probably it is not that common, or these new APIs are not super-well documented for now.

I wanted to use TableEnvironment.fromValues however, I could not use it as the test was hanging... I want to understand why and, in case, file an issue for that.

[affo]

@XComp Hello!

Final touches done and your comments are addressed 👍 I added the capability for FromElementsSource to accept a ElementsSupplier at construction phase.

The problem for which I had to implement a Serializable extend RowData was due to the fact that the FromElementsSource had a field List<OUT> elements, where OUT can also be non-serializable (which is the case of RowData), so, when the job was starting the operator couldn't be serialized.

I made it accept a ElementsSupplier extends Serializable so that it is clear that the supplier should be serializable.

In my use case, I simply made the quirk that I preserved the previous implementation using Row (which is serializable) and just convert it to RowData on get. No class has now RowData fields that prevent their serializability.

[XComp]

CI test failure is unrelated: FLINK-34513

[XComp]

@flinkbot run azure

[XComp]

I'm not gonna wait for another CI round. Looks like the CI bot didn't pick up the rerun command. Anyway, I verified that the test ran (see logs).

[XComp]

One final thing: I wasn't able to do it myself somehow. Can you change the commit message prefix from [refactor] to [FLINK-20398]? "refactor" isn't a prefix the Flink community usually use.

[affo]

@XComp done!

Don't worry in any case, I loved the review process. This is my first contribution and this is part of learning for next ones 🤝

[affo]

@flinkbot run azure

[XComp]

Yeah, the test failure is unrelated (FLINK-35042). Unfortunately, we missed to get it in before the feature freeze. Let's merge it after the release branch for 1.20 is created.

[affo]

Yeah, the test failure is unrelated (FLINK-35042). Unfortunately, we missed to get it in before the feature freeze. Let's merge it after the release branch for 1.20 is created.

Ah snap, gonna stay tuned on this then 👍 I will create the JIRA for the "on-the-fly" processing after merging as stated above 👍

[XComp]

I merged the change since release-1.20 branch is created since Tuesday. Thanks for your contribution, @affo

[affo]

@XComp thanks for your prompt reviews! Going to proceed with creating the JIRA as we agreed above 👍