clickhouse-sink-connector
clickhouse-sink-connector copied to clipboard
type conversion: timestamp to DateTime64(6)
For auto-created tables Sink Connector converts MySQL timestamp column to DateTime64(6).
The TIMESTAMP data type is used for values that contain both date and time parts. TIMESTAMP has a range of '1970-01-01 00:00:01' UTC to '2038-01-19 03:14:07' UTC.
It Should be DateTime (32 bit) type in Clickhouse.
MySQL:
`last_modify` timestamp NULL DEFAULT NULL,
Clickhouse:
last_modify Nullable(DateTime64(6)),
Agreed, it does not lose information though. DateTime64(0)
is also acceptable.
python db_load/mysql_parser/mysql_parser.py test2.sql
2023-11-11 21:57:14,279 - root - MainThread - INFO - source = create table t( id int primary key,
`last_modify` timestamp NULL DEFAULT NULL
) engine=InnoDB;
2023-11-11 21:57:14,633 - root - MainThread - INFO - {'column_name': 'id', 'datatype': 'int', 'nullable': True, 'mysql_datatype': 'int'}
2023-11-11 21:57:14,633 - root - MainThread - INFO - {'column_name': '`last_modify`', 'datatype': 'DateTime64(0)', 'nullable': True, 'mysql_datatype': 'timestamp'}
2023-11-11 21:57:14,633 - root - MainThread - INFO - CREATE TABLE t (id int NULL,
`last_modify` DateTime64(0) NULL,
`_version` UInt64 DEFAULT 0,
`is_deleted` UInt8 DEFAULT 0
) engine=ReplacingMergeTree(_version,is_deleted) order by id
2023-11-11 21:57:14,633 - root - MainThread - DEBUG - (sqlStatements (sqlStatement (ddlStatement (createTable create table (tableName (fullId (uid (simpleId t)))) (createDefinitions ( (createDefinition (fullColumnName (uid (simpleId id))) (columnDefinition (dataType int) (columnConstraint primary key))) , (createDefinition (fullColumnName (uid (simpleId (engineName `last_modify`)))) (columnDefinition (dataType timestamp) (columnConstraint (nullNotnull NULL)) (columnConstraint DEFAULT (defaultValue NULL)))) )) (tableOption engine = (engineName InnoDB))))) (emptyStatement_ ;))
2023-11-11 21:57:14,633 - root - MainThread - INFO - target = CREATE TABLE t (id int NULL,
`last_modify` DateTime64(0) NULL,
`_version` UInt64 DEFAULT 0,
`is_deleted` UInt8 DEFAULT 0
) engine=ReplacingMergeTree(_version,is_deleted) order by id
At least this behaviour is consistent with the documentation, not a bug :-)