dinky icon indicating copy to clipboard operation
dinky copied to clipboard

[Bug] [CDCSOURCE] DATE column of mysql source is changed to '1970-01-01' when sinking to starrocks

Open hillshuang opened this issue 3 years ago • 1 comments

Search before asking

  • [X] I had searched in the issues and found no similar issues.

What happened

flink task: EXECUTE CDCSOURCE jobname WITH ( 'connector' = 'mysql-cdc', 'hostname' = 'xxxxxxxxxxx', 'port' = '3306', 'username' = 'xxxxx', 'password' = 'xxxx', 'checkpoint' = '3000', 'scan.startup.mode' = 'initial', 'parallelism' = '1', 'source.server-time-zone' = 'Africa/Nairobi', 'table-name' = 'tb.repay_order', 'sink.connector' = 'starrocks', 'sink.jdbc-url' = 'jdbc:mysql://xxxxx:9030', 'sink.load-url' = 'xxxx:8030', 'sink.username' = 'flink', 'sink.password' = 'xxxx@xxxx', 'sink.sink.db' = 'kenya_biz_olap', 'sink.table.prefix' = 'ods_tb_', 'sink.table.lower' = 'true', 'sink.database-name' = '${schemaName}', 'sink.table-name' = '${tableName}', 'sink.sink.properties.format' = 'json', 'sink.sink.properties.strip_outer_array' = 'true', 'sink.sink.max-retries' = '10', 'sink.sink.buffer-flush.interval-ms' = '15000', 'sink.sink.parallelism' = '1' )

The source data of column 'end_date' (type: DATE) is '2022-10-18'
but after sinking to starrocks, it was changed to '1970-01-01'.

The data is correct when using flink tool sql-client.sh to synchronize table.

What you expected to happen

the column of type DATA should be same with source.

How to reproduce

The enviroments: Flink 1.15.3 flink-connector-starrocks-1.2.5_flink-1.15.jar flink-sql-connector-mysql-cdc-2.3.0.jar dlink-client-1.15-0.7.1.jar dlink-client-base-0.7.1.jar dlink-common-0.7.1.jar

Table of starrocks: CREATE DATABASE IF NOT EXISTS lending;

CREATE TABLE IF NOT EXISTS lending.table1 ( id INT(11) NOT NULL COMMENT "", test_date DATE NOT NULL" ) ENGINE=olap PRIMARY KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ( "replication_num" = "3" );

Anything else

No response

Version

dev

Are you willing to submit PR?

  • [ ] Yes I am willing to submit a PR!

Code of Conduct

hillshuang avatar Feb 24 '23 10:02 hillshuang

Check the date format in the source database (MySQL CDC) and the target database (Starrocks) and ensure that they are the same. If they are different, try to change the date format in the sink database to match the source format.

MatsMurmann avatar Feb 24 '23 16:02 MatsMurmann

Finally I found that with Flink: 1.15.3 JDK: 11, Date column is incorrect (1970-01-01), with Flink: 1.14.5 JDK: 1.8, Date column is correct!

hillshuang avatar Mar 01 '23 16:03 hillshuang