[tdsql] The first draft of TDSQL-CDC

[guixin]

TDSQL is an abbreviation for TecentDistributed SQL. It is derived from MySQL and designed like mysql cluster.

TDSQL CDC base on mysql cdc. It discovers tdsql's set and assigns them's splits to explicit readers for consumption. Fetcher Manager support multiple split reader thread when TM slot size lower then tdsql cluster set size.

TDSQL Document Link: https://cloud.tencent.com/document/product/557

[guixin]

@leonardBang Please take a look this pr. Thanks

[gong]

TDSQLhave TDSQL-C and TDSQL distribute mysql version. TDSQL distribute mysql version create table need shardkey. debezium will cause throw exception when using mysql-cdc-connector

create table mytable( 
data_key int,
name varchar(255),
age int,
 PRIMARY KEY (`data_key`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8  shardkey=data_key': io.debezium.text.ParsingException: DDL 
mismatched input 'shardkey' expecting {<EOF>, '--'}

ERROR io.debezium.pipeline.ErrorHandler [] - Producer failure
io.debezium.DebeziumException: Proxy ERROR: This type of SQL is not supported by group shard Error code: 602; SQLSTATE: HY000.
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1146) 

So, can this connector solve this problem?

[guixin]

TDSQLhave TDSQL-C and TDSQL distribute mysql version. TDSQL distribute mysql version create table need shardkey. debezium will cause throw exception when using mysql-cdc-connector

create table mytable( 
data_key int,
name varchar(255),
age int,
 PRIMARY KEY (`data_key`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8  shardkey=data_key': io.debezium.text.ParsingException: DDL 
mismatched input 'shardkey' expecting {<EOF>, '--'}

ERROR io.debezium.pipeline.ErrorHandler [] - Producer failure
io.debezium.DebeziumException: Proxy ERROR: This type of SQL is not supported by group shard Error code: 602; SQLSTATE: HY000.
	at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1146) 

So, can this connector solve this problem?

Yes, tdsql cdc connector solve this problem. Welcome to try tdsql cdc connector. You need clone source to local, checkout feature/tdsql-cdc branch and compile it. If you encounter problems, please create issue to me, I will try solve it.

[guixin]

TDSQLhave TDSQL-C and TDSQL distribute mysql version. TDSQL distribute mysql version create table need shardkey. debezium will cause throw exception when using mysql-cdc-connector

create table mytable( 
data_key int,
name varchar(255),
age int,
 PRIMARY KEY (`data_key`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8  shardkey=data_key': io.debezium.text.ParsingException: DDL 
mismatched input 'shardkey' expecting {<EOF>, '--'}

ERROR io.debezium.pipeline.ErrorHandler [] - Producer failure
io.debezium.DebeziumException: Proxy ERROR: This type of SQL is not supported by group shard Error code: 602; SQLSTATE: HY000.
 at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.wrap(MySqlStreamingChangeEventSource.java:1146) 

So, can this connector solve this problem?

Yes, tdsql cdc connector solve this problem. Welcome to try tdsql cdc connector. You need clone source to local, checkout feature/tdsql-cdc branch and compile it. If you encounter problems, please create issue to me, I will try solve it.

tdsql cdc table ddl schema just like mysql cdc. the example is

Flink SQL> CREATE TABLE orders (
     order_id INT,
     order_date TIMESTAMP(0),
     customer_name STRING,
     price DECIMAL(10, 5),
     product_id INT,
     order_status BOOLEAN,
     PRIMARY KEY(order_id) NOT ENFORCED
     ) WITH (
     'connector' = 'tdsql-cdc',
     'hostname' = 'localhost',
     'port' = '3306',
     'username' = 'root',
     'password' = '123456',
     'database-name' = 'mydb',
     'table-name' = 'orders');

[yuxiqian]

Hi @guixin, Thanks for your great work! Also sorry for the delay. Could you please rebase it to latest master branch since there’s been lots of changes in Flink CDC repo since your original commit? Kindly reminder that com.ververica.cdc.connectors package has been moved to org.apache.flink.cdc.connectors, and your new connector should be placed under flink-cdc-connect/flink-cdc-source-connectors path.

[guixin]

Hi @guixin, Thanks for your great work! Also sorry for the delay. Could you please rebase it to latest master branch since there’s been lots of changes in Flink CDC repo since your original commit? Kindly reminder that com.ververica.cdc.connectors package has been moved to org.apache.flink.cdc.connectors, and your new connector should be placed under flink-cdc-connect/flink-cdc-source-connectors path.

Thanks for the reminder. I closed this pull request firstly