paimon icon indicating copy to clipboard operation
paimon copied to clipboard

[Bug] MySQL add column sync Paimon failed

Open DavidZ1 opened this issue 3 months ago • 1 comments

Search before asking

  • [x] I searched in the issues and found nothing similar.

Paimon version

1.0.1

Compute Engine

Flink 1.18.1

Minimal reproduce step

1. started sync job

/data/module/flink-1.18.1/bin/flink run-application -d -t yarn-application \
 -p 1 \
 -Dyarn.application.queue=realtime \
 -Dyarn.application.name=paimon-schema-test \
 -Djobmanager.memory.process.size=1024mb \
 -Dtaskmanager.memory.process.size=2048mb \
 -Dtaskmanager.memory.network.max=64mb \
 -Dtaskmanager.numberOfTaskSlots=1 \
 -Dexecution.checkpointing.interval=60000 \
  -c org.apache.paimon.flink.action.FlinkActions \
  /paimon/paimon-flink-action-1.0.1.jar \
  mysql_sync_database \
  --mode combined \
  --warehouse hdfs:///hive/paimon/  \
  --database tmp  \
  --table_prefix "ods_"  \
  --mysql_conf hostname=xxx \
  --mysql_conf username=xxx \
  --mysql_conf password=xxx \
  --mysql_conf database-name=test   \
  --catalog_conf metastore=hive \
  --catalog_conf uri=thrift://xxx:7004,thrift://xxx:7004  \
  --table_conf bucket=1  \
  --table_conf changelog-producer=input \
  --type_mapping tinyint1-not-bool,char-to-string \
  --including_tables 'add_colum_table_test'

2. Dbeaver execute ddl

alter table test.add_colum_table_test add column city_name varchar(32) NOT null default '0' COMMENT '城市名称';

3. paimon table sync job fail

2025-09-16 10:12:55
java.lang.IllegalArgumentException: Column city_name cannot specify NOT NULL in the test.add_colum_table_test  table.
	at org.apache.paimon.utils.Preconditions.checkArgument(Preconditions.java:149)
	at org.apache.paimon.schema.SchemaManager.commitChanges(SchemaManager.java:333)

Why SchemaManager.java has check add column check not null ?

What doesn't meet your expectations?

paimon table sync job fail

2025-09-16 10:12:55
java.lang.IllegalArgumentException: Column city_name cannot specify NOT NULL in the test.add_colum_table_test  table.
	at org.apache.paimon.utils.Preconditions.checkArgument(Preconditions.java:149)
	at org.apache.paimon.schema.SchemaManager.commitChanges(SchemaManager.java:333)
	at org.apache.paimon.hive.HiveCatalog.alterTableImpl(HiveCatalog.java:905)
	at org.apache.paimon.catalog.AbstractCatalog.alterTable(AbstractCatalog.java:375)
	at org.apache.paimon.catalog.Catalog.alterTable(Catalog.java:278)
	at org.apache.paimon.flink.sink.cdc.UpdatedDataFieldsProcessFunctionBase.applySchemaChange(UpdatedDataFieldsProcessFunctionBase.java:98)
	at org.apache.paimon.flink.sink.cdc.MultiTableUpdatedDataFieldsProcessFunction.processElement(MultiTableUpdatedDataFieldsProcessFunction.java:84)
	at org.apache.paimon.flink.sink.cdc.MultiTableUpdatedDataFieldsProcessFunction.processElement(MultiTableUpdatedDataFieldsProcessFunction.java:47)
	at org.apache.flink.streaming.api.operators.LegacyKeyedProcessOperator.processElement(LegacyKeyedProcessOperator.java:88)
	at org.apache.flink.streaming.runtime.io.RecordProcessorUtils.lambda$getRecordProcessor$0(RecordProcessorUtils.java:60)
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:237)
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:146)
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:562)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:858)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:807)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:932)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
	at java.lang.Thread.run(Thread.java:748)

Anything else?

No response

Are you willing to submit a PR?

  • [ ] I'm willing to submit a PR!

DavidZ1 avatar Sep 16 '25 02:09 DavidZ1

@DavidZ1 It should be that the columns of the paimon table do not support default values.

sd4324530 avatar Oct 11 '25 05:10 sd4324530