apache
[vitess] Add vitess cdc connector
This is initial implementation of Vitess CDC connector. For a fully detailed documentation of Vitess Debezium Connector , please see Documentation.
Thanks @s-gelazevicius for the great contribution, I'm not familiar with Vitess, but I'm willing to learn it. Could you share more about the background of vitess-cdc connector? e.g. How is it used in your business/company, the data-scale etc.
Hi @leonardBang
Thanks @s-gelazevicius for the great contribution, I'm not familiar with Vitess, but I'm willing to learn it. Could you share more about the background of vitess-cdc connector? e.g. How is it used in your business/company, the data-scale etc.
Vitess is clustering solution on top of Mysql. It automatically handles sharding and has some other nice features. From client perspective same Mysql JDBC driver can be used.
Here in Vinted, databases are quite large and we used Mysql directly, but it was hard to scale, sharding required a lot of manual work. So now we are migrating to Vitess.
This connector will be used for our CDC ingestion pipeline implemented as a flink application.
Hi, @s-gelazevicius
Here in Vinted, databases are quite large and we used Mysql directly, but it was hard to scale, sharding required a lot of manual work. So now we are migrating to Vitess. This connector will be used for our CDC ingestion pipeline implemented as a flink application.
Thanks for your detailed explanation, happy to see the developer from Vinted and welcome to community, I'll help review this PR.
Hi, @leonardBang any updates on this pull request. I have re-based it from the latest master to keep it fresh. It's been siting here for a long time. It would be really nice to have this in upstream.
@s-gelazevicius Thanks for the great work and BIG sorry for my late response. I'm busy on supporting the TiDB and OceanBase vendors, I'll review this PR soon after the OceanBase PR review finished.
@fsk119 thank you for review. I already addressed some of your comments, and will fix other issues during next week.
Hey @s-gelazevicius, thanks a lot for the awesome work on this Vitess connector! As we are approaching the feature freeze of Flink CDC version 2.4.0, we are going to make some patches on your current work and merge your commits to master. We'll preserve all your changes and keep the commit under your name and email address 😄
I squashed all commits into one to keep the history of master branch clean. The CI is running in my own Azure pipeline. I'll merge the commit into master once the CI is green.
~~The only thing we are still missing for the new Vitess connector is the end-to-end test case. Could you add one under flink-cdc-e2e-tests in another PR when you are available? Thanks!~~ I added E2E cases in order to make the release more stable.
Hey @s-gelazevicius, thanks a lot for the awesome work on this Vitess connector! As we are approaching the feature freeze of Flink CDC version 2.4.0, we are going to make some patches on your current work and merge your commits to master. We'll preserve all your changes and keep the commit under your name and email address 😄
I squashed all commits into one to keep the history of master branch clean. The CI is running in my own Azure pipeline. I'll merge the commit into master once the CI is green.
The only thing we are still missing for the new Vitess connector is the end-to-end test case. Could you add one under
flink-cdc-e2e-testsin another PR when you are available? Thanks!
Thanks @s-gelazevicius and @PatrickRen for the contribution,
Could we keep both gintarasm and s-gelazevicius 's contributions when squash the commits? and I think that we should add an e2e test for new joined connector.