[feature](cdc)add ignoreIncompatible option

[vinlee19]

Proposed changes

1. In previous versions, encountering a source schema that did not match Doris would lead to the failure of the entire job. This was especially problematic when dealing with source instances containing thousands of tables. Moreover, adding a new table to the source database with a schema incompatible with Doris would also cause the job to fail. To address this issue, we introduced the --ignore-incompatible=true/false option. When set to true, incompatible schemas are ignored, whereas setting it to false triggers an error for mismatched schemas. The default value of --ignore-incompatible is true.

For example, we create tables AA\C and #1234 in Oracle, but their schemas do not match those in Doris.

We run in the Flink CLI in the following way:

<FLINK_HOME>bin/flink run \
    -Dexecution.checkpointing.interval=10s \
    -Dparallelism.default=1 \
    -c org.apache.doris.flink.tools.cdc.CdcTools \
    lib/flink-doris-connector-1.17-1.6.0-SNAPSHOT.jar \
    mysql-sync-database \
    --database test_db \
    --mysql-conf hostname=127.0.0.1 \
    --mysql-conf port=3306 \
    --mysql-conf username=root \
    --mysql-conf password=123456 \
    --mysql-conf database-name=mysql_db \
    --including-tables "tbl1|test.*" \
    --sink-conf fenodes=127.0.0.1:8030 \
    --sink-conf username=root \
    --sink-conf password=123456 \
    --sink-conf jdbc-url=jdbc:mysql://127.0.0.1:9030 \
    --sink-conf sink.label-prefix=label \
    --table-conf replication_num=1  \
    --single-sink "true" \
   --use-new-schema-change "true" \
    --ignore-incompatible=false 

3. We have refactored the DorisSystem createTable DDL code. Issue Number: close #xxx

Problem Summary:

Describe the overview of changes.

Checklist(Required)

1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

[JNSimba]

You used params.has("ignore-incompatible"); , so there is no distinction between true and false. As long as it appears, it is true, and if it is not written, it is false.

[vinlee19]

@JNSimba I have changed the approach for retrieving the parameters ignore-incompatible.