apache
[FLINK-36520] CDC Supports MySQL 8.x and above
This closes FLINK-36520, FLINK-36590.
Currently, trying to use MySQL CDC with MySQL 8.4 would cause the following exception:
Caused by: org.apache.flink.util.FlinkRuntimeException: Read split MySqlSnapshotSplit{tableId=customer.customers, splitId='customer.customers:0', splitKeyType=[`id` BIGINT NOT NULL], splitStart=null, splitEnd=null, highWatermark=null} error due to org.apache.flink.util.FlinkRuntimeException: Cannot read the binlog filename and position via 'SHOW MASTER STATUS'. Make sure your server is correctly configured.
The reason is MySQL has changed some questionable words like MASTER and SLAVE to more appropriate ones, so some commands are no longer compatible. Legacy MySqlSource (based on Debezium) wasn't fixed since it's not possible to upgrade Debezium dependency for now, and it has been deprecated for a long time anyway.
Marked this PR as a draft, needs more discussion and testing over this.
Could you investigate binlog compress feature would been supported via this PR or not? It requires MySQL 8.0.20 above version IIRC.
Correct me if I'm wrong, but it looks like using the proper SQL command for showing binary logs is the only major change here. The rest is about refactoring the test suite.
@leonardBang is the support for binlog compression a blocker for these changes to get merged?
[...] needs more discussion and testing over this.
@yuxiqian do you have any specific discussion topics in mind? As for testing, I believe having the test suite pass would be a great start, although it's not clear why the CI didn't run the tests on this PR.
Thanks @morozov's kindly review!
@leonardBang is the support for binlog compression a blocker for these changes to get merged?
I did some experiment earlier, and noticed that binlog compression would break current binlog backfilling code. However, considering this feature was introduced in MySQL 8.0.20, perhaps it could be handled independently.
@yuxiqian do you have any specific discussion topics in mind? As for testing, I believe having the test suite pass would be a great start, although it's not clear why the CI didn't run the tests on this PR.
Seems there are a lot of merging conflicts blocking the CI. Will resolve them soon.
A release of this would be very helpful as MySQL 8.0 is nearing it's end of life.
E.g. AWS stops supporting version 8.0 soon.
Hi, I would like to ask why we do not use Debezium directly, but implement it ourselves? Debezium already supports MySQL 8 and above, and Debezium updates much faster than we do.
Hi, I would like to ask why we do not use Debezium directly, but implement it ourselves? Debezium already supports MySQL 8 and above, and Debezium updates much faster than we do.
Thanks for @shengjk's nice suggestion! For a long time we've been stuck at Debezium 1.9.x and monkey-patching Debezium, mainly for keeping compatibility with Java 8.
FLINK-36605 also suggests a similar idea, but it is more likely to happen after bumping to Flink 2.x and dropping Java 8 support.
@yuxiqian Hi, Is there any problem? Why is this PR in draft status? Does it have an expected merge date?
@yuxiqian Hi, Is there any problem? Why is this PR in draft status? Does it have an expected merge date?
Thanks @zhangshenghang for tracing this issue!
IIRC, one concern from maintainers is we can't provide decent support for some newly introduced MySQL features, like Binlog compression, VECTOR data types, etc., without upgrading Debezium dependencies, which requires dropping Java 8 and Flink 1.x support. So this PR is, at best, a transitional solution.
The next feature release (3.6 or 4.0) has not been kicked-off, but there's some discussions on upgrading Debezium versions here (https://issues.apache.org/jira/browse/FLINK-36605), so I'm not sure if it's appropriate to introduce this temporary change at this time.
Anyway, I'll try to resolve all conflicts later this week so it could be marked ready for review.
@yuxiqian Hi, Is there any problem? Why is this PR in draft status? Does it have an expected merge date?
Thanks @zhangshenghang for tracing this issue!
IIRC, one concern from maintainers is we can't provide descent support for some newly introduced MySQL features, like Binlog compression, VECTOR data types, etc., without upgrading Debezium dependencies, which requires dropping Java 8 and Flink 1.x support. So this PR is, at best, a transitional solution.
The next feature release (3.6 or 4.0) has not been kicked-off, but there's some discussions on upgrading Debezium versions here (https://issues.apache.org/jira/browse/FLINK-36605), so I'm not sure if it's appropriate to introduce this temporary change at this time.
Anyway, I'll try to resolve all conflicts later this week so it could be marked ready for review.
@yuxiqian Thank you for your reply. Yes, upgrading to JDK 11 would be a significant change. If there could be a temporary transitional solution for 1.8, that would also be a good option.
@yuxiqian Hi, Is there any problem? Why is this PR in draft status? Does it have an expected merge date?
Thanks @zhangshenghang for tracing this issue! IIRC, one concern from maintainers is we can't provide descent support for some newly introduced MySQL features, like Binlog compression, VECTOR data types, etc., without upgrading Debezium dependencies, which requires dropping Java 8 and Flink 1.x support. So this PR is, at best, a transitional solution. The next feature release (3.6 or 4.0) has not been kicked-off, but there's some discussions on upgrading Debezium versions here (https://issues.apache.org/jira/browse/FLINK-36605), so I'm not sure if it's appropriate to introduce this temporary change at this time. Anyway, I'll try to resolve all conflicts later this week so it could be marked ready for review.
@yuxiqian Thank you for your reply. Yes, upgrading to JDK 11 would be a significant change. If there could be a temporary transitional solution for 1.8, that would also be a good option.
@yuxiqian Hi，I saw a discussion here about upgrading Debezium to version 2.7.x: https://issues.apache.org/jira/browse/FLINK-36605.
However, it seems that version 2.7.x also does not support MySQL 8.4.
MySQL 8.4 supported starting from version 3.2
Thanks @zhangshenghang for confirming this! Debezium 3.2 requiring Java 17 seems not very likely, since Flink 2.x still supports Java 11 (deprecated though). I'll fix CI test cases first.