[Bug] first ddl with multi-column changes function will not be executed

[icchux]

Search before asking

• [X] I had searched in the issues and found no similar issues.

Version

master branch

<revision>1.5.0-SNAPSHOT</revision>
<flink.version>1.16.0</flink.version>
<flink.minor.version>1.16</flink.minor.version>
<flink.sql.cdc.version>2.4.1</flink.sql.cdc.version>

What's Wrong?

mysql-sync-database -use-new-schema-change when the program starts for the first time, firstSchemeChange==true will initialize the originFieldSchema Map, but the number of fields in updateFiledSchema is the same as that in originFieldSchema Map, no subsequent ddl changes will be made. the main method follow

What You Expected?

expected it do correct at the first schema change after job start

How to Reproduce?

and Here are my execution steps： 1.

    add state_10 tinyint null comment '状态';

sync schema change failed : INFO org.apache.doris.flink.sink.writer.JsonDebeziumSchemaSerializer [Source: MySQL Source -> Process -> test3: Writer -> test3: Committer (1/10)#0] - ddl can not do schema change: ***

alter table test3
    add state_11 tinyint null comment '状态'

sync schema change success : INFO org.apache.doris.flink.sink.writer.JsonDebeziumSchemaSerializer [Source: MySQL Source -> Process -> test3: Writer -> test3: Committer (1/10)#0] - schema change status:true, ddl:ALTER TABLE test.test3 ADD COLUMN state_11 TINYINT

Anything Else?

No response

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[icchux]

Hope to provide assistance

[DongLiang-0]

@icchux In this first schema change, in which scenario will the initialization field and the changed field be the same? Can you provide the corresponding table structure and detailed reproduction steps?

[icchux]

@icchux In this first schema change, in which scenario will the initialization field and the changed field be the same? Can you provide the corresponding table structure and detailed reproduction steps?

@DongLiang-0 get. the reproduction steps follow:

1. mysql create table :

CREATE TABLE `test1` (
                         `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '自增主键',
                         `state` tinyint(4) DEFAULT NULL COMMENT '状态',
                         `is_finish` tinyint(4) DEFAULT NULL COMMENT '是否完成',
                         PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='测试表';

2. use the master branch run main method org.apache.doris.flink.tools.cdc.CdcTools with parameters :

mysql-sync-database
--database test
--use-new-schema-change
--mysql-conf hostname=***
--mysql-conf port=3306
--mysql-conf username=***
--mysql-conf password=***
--mysql-conf database-name=***
--mysql-conf server-time-zone=UTC
--including-tables "test1"
--sink-conf fenodes=192.168.6.69:8030
--sink-conf username=root
--sink-conf password=***
--sink-conf jdbc-url=jdbc:mysql://192.168.6.69:9030
--sink-conf sink.label-prefix=label
--table-conf replication_num=1

3. mysql execute first ddl

alter table test1
    add state_1 tinyint null comment '状态1',
    add state_2 tinyint null comment '状态2';

the doris won't execute this ddl.

4. mysql execute second ddl

alter table test1
    add state_3 tinyint null comment '状态3',
    add state_4 tinyint null comment '状态4';

this time doris will sync execute this ddl.

[DongLiang-0]

Hi, I followed the steps you mentioned exactly and did not reproduce the problem described above. @icchux

[DongLiang-0]

I guess the situation you described above is caused by the corresponding extractDDLList method being executed twice when you were debugging? But this will not happen in actual production.

[icchux]

I guess the situation you described above is caused by the corresponding extractDDLList method being executed twice when you were debugging? But this will not happen in actual production.

ok thanks , it looks well. maybe it's my own problem, I will try again to reposition the specific issue

[DongLiang-0]

Thank you for raising this issue. I confirmed that this problem does exist here. This problem occurs when the data in the table is empty.