flink icon indicating copy to clipboard operation
flink copied to clipboard

[WIP][FLINK-28433][connector/jdbc]Add mariadb jdbc connection validation

Open bzhaoopenstack opened this issue 2 years ago • 13 comments

What is the purpose of the change

  1. Flink Connector support the jdbc connection with Mysql series DB, such as MySQL, MariaDB, Percona and etc. But we see the doc reference[1] shows the Mariadb Connector/J 3.0 will accepts jdbc:mariadb: as the protocal in connection strings by default. So we need to apply this change in our mysql jdbc connection vailidation. Mariadb Connector/J 2.X still support both "jdbc:mysql:" and "jdbc:mariadb:".
  2. Introduce MariaDB dialect implemented like Mysql.
  3. Add UTs for DataTypes and a testcontainer for MariaDB in JdbcExactlyOnceSinkE2eTest.

[1] https://mariadb.com/kb/en/about-mariadb-connector-j/#jdbcmysql-scheme-compatibility

Brief change log

  • Making Mysql jdbc connector validate the MariaDB type connection protocol for fit users to use Mysql jdbc connector to access MariaDB instances.
  • Introduce MariaDB jdbc dialect into Flink, which is very similar with Mysql.
  • Introduce UTs and E2E test for them.

Verifying this change

Exec mvn clean package -Dfast -Pskip-webui-build -pl flink-connectors/flink-connector-jdbc Or mvn clean package -Dfast -Pskip-webui-build -pl flink-connectors/flink-connector-jdbc -Dtest="org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest" for run all tests or a single test. Notes that all types didn't cover, and will be on going for support.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (docs)

bzhaoopenstack avatar Jul 07 '22 07:07 bzhaoopenstack

CI report:

  • ecea05c4a1e052366570f864b3785dee25e0810b Azure: SUCCESS
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Jul 07 '22 07:07 flinkbot

+1, After the change. We can use mariadb driver to connect the mysql through by setting the jdbcOptions of driver parameter without change the url prefix.

Peng-Lei avatar Jul 07 '22 07:07 Peng-Lei

@bzhaoopenstack Thanks for the PR, but in order to add support for MariaDB I would also like to see a test and documentation in place.

Thanks very much for reply and suggest. I will add the test and documentation. Once I'm finish, I think I need your kind review and advice. Thank you. ;-)

bzhaoopenstack avatar Jul 08 '22 00:07 bzhaoopenstack

@bzhaoopenstack Thanks for the PR, but in order to add support for MariaDB I would also like to see a test and documentation in place.

Hi @MartijnVisser, I have a question here, you said you want a test for this PR, does it mean we should add a specific UT which only verify the jdbc conncetion strings? Because I didn't find any existing UT in Flink. And notes: We only use Mysql connector to connect MariaDB, as it is a Mysql-series database.

bzhaoopenstack avatar Jul 08 '22 08:07 bzhaoopenstack

I would expect that we add MariaDB to the JdbcExactlyOnceSinkE2eTest with the correct test container and have a MariaDB dialect implemented (even if that would be just a copy of MySQL)

MartijnVisser avatar Jul 08 '22 11:07 MartijnVisser

I would expect that we add MariaDB to the JdbcExactlyOnceSinkE2eTest with the correct test container and have a MariaDB dialect implemented (even if that would be just a copy of MySQL)

Thanks for explain more, this is a good direction/suggestion for me.

bzhaoopenstack avatar Jul 11 '22 00:07 bzhaoopenstack

Hi @MartijnVisser , sorry for late. And this is my first PR in flink, so I wish you could leave some kind directions and suggestions. Thank you very much. ;-). And hope this doesn't disturb you so much.

Now, I will request an new review from your side, once we both think this PR is ready, I will remove the title [WIP]. ;-)

bzhaoopenstack avatar Jul 12 '22 09:07 bzhaoopenstack

@afedulov Do you want to have a look at this one?

MartijnVisser avatar Jul 13 '22 15:07 MartijnVisser

Hi all. @MartijnVisser @hadronzoo any updates on this one? Thanks

bzhaoopenstack avatar Jul 18 '22 01:07 bzhaoopenstack

@bzhaoopenstack Are you ready?If yes, please change the title without "WIP".

Peng-Lei avatar Jul 19 '22 08:07 Peng-Lei

@bzhaoopenstack Are you ready?If yes, please change the title without "WIP".

Yeah, I'm ready for the kind suggest from @MartijnVisser and @hadronzoo . ;-)

bzhaoopenstack avatar Jul 19 '22 11:07 bzhaoopenstack

@bzhaoopenstack I won't be able to review this at the moment, but keep in mind that the CI is still failing for this PR. It makes sense to make sure that the CI works before the review actually starts.

MartijnVisser avatar Aug 01 '22 13:08 MartijnVisser

@bzhaoopenstack I won't be able to review this at the moment, but keep in mind that the CI is still failing for this PR. It makes sense to make sure that the CI works before the review actually starts.

OK, thank you Martijn. I will resolve it and make it ready for review

bzhaoopenstack avatar Aug 01 '22 14:08 bzhaoopenstack

Sorry for late update. ;-) . Let's wait the azure test result. I got pass on my local env.

bzhaoopenstack avatar Aug 22 '22 03:08 bzhaoopenstack

@flinkbot run azure

bzhaoopenstack avatar Aug 24 '22 04:08 bzhaoopenstack

We've now moved the code from the JDBC connector to its own dedicated repository; please re-route this PR to https://github.com/apache/flink-connector-jdbc

MartijnVisser avatar Nov 29 '22 14:11 MartijnVisser