paimon
paimon copied to clipboard
[Bug] MySQL add column sync Paimon failed
Search before asking
- [x] I searched in the issues and found nothing similar.
Paimon version
1.0.1
Compute Engine
Flink 1.18.1
Minimal reproduce step
1. started sync job
/data/module/flink-1.18.1/bin/flink run-application -d -t yarn-application \
-p 1 \
-Dyarn.application.queue=realtime \
-Dyarn.application.name=paimon-schema-test \
-Djobmanager.memory.process.size=1024mb \
-Dtaskmanager.memory.process.size=2048mb \
-Dtaskmanager.memory.network.max=64mb \
-Dtaskmanager.numberOfTaskSlots=1 \
-Dexecution.checkpointing.interval=60000 \
-c org.apache.paimon.flink.action.FlinkActions \
/paimon/paimon-flink-action-1.0.1.jar \
mysql_sync_database \
--mode combined \
--warehouse hdfs:///hive/paimon/ \
--database tmp \
--table_prefix "ods_" \
--mysql_conf hostname=xxx \
--mysql_conf username=xxx \
--mysql_conf password=xxx \
--mysql_conf database-name=test \
--catalog_conf metastore=hive \
--catalog_conf uri=thrift://xxx:7004,thrift://xxx:7004 \
--table_conf bucket=1 \
--table_conf changelog-producer=input \
--type_mapping tinyint1-not-bool,char-to-string \
--including_tables 'add_colum_table_test'
2. Dbeaver execute ddl
alter table test.add_colum_table_test add column city_name varchar(32) NOT null default '0' COMMENT '城市名称';
3. paimon table sync job fail
2025-09-16 10:12:55
java.lang.IllegalArgumentException: Column city_name cannot specify NOT NULL in the test.add_colum_table_test table.
at org.apache.paimon.utils.Preconditions.checkArgument(Preconditions.java:149)
at org.apache.paimon.schema.SchemaManager.commitChanges(SchemaManager.java:333)
Why SchemaManager.java has check add column check not null ?
What doesn't meet your expectations?
paimon table sync job fail
2025-09-16 10:12:55
java.lang.IllegalArgumentException: Column city_name cannot specify NOT NULL in the test.add_colum_table_test table.
at org.apache.paimon.utils.Preconditions.checkArgument(Preconditions.java:149)
at org.apache.paimon.schema.SchemaManager.commitChanges(SchemaManager.java:333)
at org.apache.paimon.hive.HiveCatalog.alterTableImpl(HiveCatalog.java:905)
at org.apache.paimon.catalog.AbstractCatalog.alterTable(AbstractCatalog.java:375)
at org.apache.paimon.catalog.Catalog.alterTable(Catalog.java:278)
at org.apache.paimon.flink.sink.cdc.UpdatedDataFieldsProcessFunctionBase.applySchemaChange(UpdatedDataFieldsProcessFunctionBase.java:98)
at org.apache.paimon.flink.sink.cdc.MultiTableUpdatedDataFieldsProcessFunction.processElement(MultiTableUpdatedDataFieldsProcessFunction.java:84)
at org.apache.paimon.flink.sink.cdc.MultiTableUpdatedDataFieldsProcessFunction.processElement(MultiTableUpdatedDataFieldsProcessFunction.java:47)
at org.apache.flink.streaming.api.operators.LegacyKeyedProcessOperator.processElement(LegacyKeyedProcessOperator.java:88)
at org.apache.flink.streaming.runtime.io.RecordProcessorUtils.lambda$getRecordProcessor$0(RecordProcessorUtils.java:60)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:237)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:146)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:562)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:858)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:807)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:932)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
at java.lang.Thread.run(Thread.java:748)
Anything else?
No response
Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
@DavidZ1 It should be that the columns of the paimon table do not support default values.