doris-flink-connector icon indicating copy to clipboard operation
doris-flink-connector copied to clipboard

[fix](cdc)fix the error mapping default values from RDBMS to Doris

Open vinlee19 opened this issue 1 year ago • 0 comments

Proposed changes

Issue Number: close #xxx

Problem Summary:

Currently, Doris only supports current_timestamp as the default function. In relational databases, the composition of different default values varies. Here are some examples: 1.MySQL In MySQL, default values for numeric types are represented as 1.0 or 0, while for string and date types, they are enclosed in single quotes. In MySQL5.7.x,you can create SQL statements using the following syntax:

create table test_str
(
    id         int auto_increment primary key,
    t2_varchar varchar(10) default '''doris'''''   null,
    t1_varchar varchar(10) default 'doris'''       null,
    t3_varchar varchar(20) default '''''doris''''' null,
    t_int int default 1,
   t_float default 1.1
);

In MySQL8.0.x,you can create table with default function.

create table t1
(
    i int auto_increment
        primary key,
    c varchar(10) default ''                              null,
    f float       default ((rand() * rand()))             null,
    b binary(16)  default (uuid_to_bin(uuid()))           null,
    d date        default ((curdate() + interval 1 year)) null,
    j json        default (json_array())                  null
);

2.Oracle In Oracle, character types are enclosed in single quotes, while the default values for timestamps and dates are obtained through the TO_TIMESTAMP and TO_DATE functions, respectively. Here are some specific examples:

create table TEST_ALL_TYPES
(
    ID              NUMBER not null constraint TEST_ALL_TYPES_PK primary key,
    N1              NUMBER,
    NUM2        NUMBER(5, -2)                default (100 / 3),
    NUM4        NUMBER(5, 7),
    T1              DATE                         default to_date('2010-2-12 10:20:30', 'YYYY-MM-DD HH24:MI:SS'),
    T2            TIMESTAMP(7)                 default TO_TIMESTAMP('2023-01-01 00:00:00.0000001'

3.PostgreSQL In PostgreSQL, default values for character and numeric types are composed of the value itself, followed by :: and the type.

create table test_all_types
(
    id              integer        default 1 not null,
    char_value      char(100)      default 'doris'::bpchar,
    varchar_value   varchar(128)   default 'make doris greate'::character varying,
    date_value      date           default '2024-01-01'::date,
    smallint_value  smallint       default 12,
    int_value       integer        default 100,
    bigint_value    bigint         default 10000,
    timestamp_value timestamp      default '2024-01-01 01:01:01.000001'::timestamp without time zone,
    decimal_value   numeric(10, 3) default 1.23,
    bit_value       bit,
    ts_decimal      numeric(38)    default 12,
    text_value      text           default 'make doris greate'::text
);

4.SQLSever In SQL Server, numeric types are enclosed in double parentheses (), while string and time types are enclosed in single parentheses ().

create table all_type
(
    id                   int not null primary key,
    name                 varchar(10)     default 'kevin',
    age                      int             default 1,
    smalldatetime_value  smalldatetime   default '2024-01-01 00:00:00.00000001',
    datetimeoffset_value datetimeoffset  default '2024-01-01 00:00:00.0000001',
    bit_value            bit             default 1,
    c1                   varchar(255)    default 12.11
)
go

In this PR, you can automatically map default values from MySQL, Oracle, PostgreSQL, and SQL Server to Doris default values. Currently, only the translation of the time default function current_timestamp is supported.

Checklist(Required)

  1. Does it affect the original behavior: (Yes/No/I Don't know)
  2. Has unit tests been added: (Yes/No/No Need)
  3. Has document been added or modified: (Yes/No/No Need)
  4. Does it need to update dependencies: (Yes/No)
  5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

vinlee19 avatar Jan 26 '24 07:01 vinlee19