flink
flink copied to clipboard
[WIP][FLINK-28433][connector/jdbc]Add mariadb jdbc connection validation
What is the purpose of the change
- Flink Connector support the jdbc connection with Mysql series DB, such as MySQL, MariaDB, Percona and etc. But we see the doc reference[1] shows the Mariadb Connector/J 3.0 will accepts jdbc:mariadb: as the protocal in connection strings by default. So we need to apply this change in our mysql jdbc connection vailidation. Mariadb Connector/J 2.X still support both "jdbc:mysql:" and "jdbc:mariadb:".
- Introduce MariaDB dialect implemented like Mysql.
- Add UTs for DataTypes and a testcontainer for MariaDB in JdbcExactlyOnceSinkE2eTest.
[1] https://mariadb.com/kb/en/about-mariadb-connector-j/#jdbcmysql-scheme-compatibility
Brief change log
- Making Mysql jdbc connector validate the MariaDB type connection protocol for fit users to use Mysql jdbc connector to access MariaDB instances.
- Introduce MariaDB jdbc dialect into Flink, which is very similar with Mysql.
- Introduce UTs and E2E test for them.
Verifying this change
Exec mvn clean package -Dfast -Pskip-webui-build -pl flink-connectors/flink-connector-jdbc
Or mvn clean package -Dfast -Pskip-webui-build -pl flink-connectors/flink-connector-jdbc -Dtest="org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest"
for run all tests or a single test.
Notes that all types didn't cover, and will be on going for support.
Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
@Public(Evolving)
: (no) - The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (docs)
CI report:
- ecea05c4a1e052366570f864b3785dee25e0810b Azure: SUCCESS
Bot commands
The @flinkbot bot supports the following commands:-
@flinkbot run azure
re-run the last Azure build
+1, After the change. We can use mariadb driver to connect the mysql through by setting the jdbcOptions of driver
parameter without change the url prefix.
@bzhaoopenstack Thanks for the PR, but in order to add support for MariaDB I would also like to see a test and documentation in place.
Thanks very much for reply and suggest. I will add the test and documentation. Once I'm finish, I think I need your kind review and advice. Thank you. ;-)
@bzhaoopenstack Thanks for the PR, but in order to add support for MariaDB I would also like to see a test and documentation in place.
Hi @MartijnVisser, I have a question here, you said you want a test for this PR, does it mean we should add a specific UT which only verify the jdbc conncetion strings? Because I didn't find any existing UT in Flink. And notes: We only use Mysql connector to connect MariaDB, as it is a Mysql-series database.
I would expect that we add MariaDB to the JdbcExactlyOnceSinkE2eTest
with the correct test container and have a MariaDB dialect implemented (even if that would be just a copy of MySQL)
I would expect that we add MariaDB to the
JdbcExactlyOnceSinkE2eTest
with the correct test container and have a MariaDB dialect implemented (even if that would be just a copy of MySQL)
Thanks for explain more, this is a good direction/suggestion for me.
Hi @MartijnVisser , sorry for late. And this is my first PR in flink, so I wish you could leave some kind directions and suggestions. Thank you very much. ;-). And hope this doesn't disturb you so much.
Now, I will request an new review from your side, once we both think this PR is ready, I will remove the title [WIP]. ;-)
@afedulov Do you want to have a look at this one?
Hi all. @MartijnVisser @hadronzoo any updates on this one? Thanks
@bzhaoopenstack Are you ready?If yes, please change the title without "WIP".
@bzhaoopenstack Are you ready?If yes, please change the title without "WIP".
Yeah, I'm ready for the kind suggest from @MartijnVisser and @hadronzoo . ;-)
@bzhaoopenstack I won't be able to review this at the moment, but keep in mind that the CI is still failing for this PR. It makes sense to make sure that the CI works before the review actually starts.
@bzhaoopenstack I won't be able to review this at the moment, but keep in mind that the CI is still failing for this PR. It makes sense to make sure that the CI works before the review actually starts.
OK, thank you Martijn. I will resolve it and make it ready for review
Sorry for late update. ;-) . Let's wait the azure test result. I got pass on my local env.
@flinkbot run azure
We've now moved the code from the JDBC connector to its own dedicated repository; please re-route this PR to https://github.com/apache/flink-connector-jdbc