flink-cdc
flink-cdc copied to clipboard
apache
Caused by: io.debezium.DebeziumException: java.lang.NullPointerException
Describe the bug(Please use English) A clear and concise description of what the bug is.
Environment :
- Flink version : 1.14.3
- Flink CDC version: 2.2.1
- Database and version: Mysql 5.7
To Reproduce Steps to reproduce the behavior:
- Thes test data :
- The test code :
- The error :
Additional Description
If applicable, add screenshots to help explain your problem.
When I initialize the whole library snapshot to read null pointer abnormal community partners encountered or have a solution to share, I read the source code did not find the reason
I also encountered the same problem, the project can not go on.
Environment :
Flink version : 1.14.5 Flink CDC version: 2.2.1 Database and version: Mysql 5.7
2022-09-13 11:24:36,662 ERROR com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader [] - Execute snapshot read task for mysql split MySqlSnapshotSplit{tableId=gc_fps_receivable.headorderdetail, splitId='gc_fps_receivable.headorderdetail:0', splitKeyType=[`HeadOrderDetailId` INT NOT NULL], splitStart=null, splitEnd=null, highWatermark=null} fail
io.debezium.DebeziumException: java.lang.NullPointerException
at com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.execute(MySqlSnapshotSplitReadTask.java:120) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
at com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader.lambda$submitSplit$0(SnapshotSplitReader.java:117) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_342]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_342]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_342]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_342]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_342]
Caused by: java.lang.NullPointerException
at io.debezium.util.ColumnUtils.toArray(ColumnUtils.java:41) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
at com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.createDataEventsForTable(MySqlSnapshotSplitReadTask.java:230) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
at com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.createDataEvents(MySqlSnapshotSplitReadTask.java:191) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
at com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.doExecute(MySqlSnapshotSplitReadTask.java:150) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
at com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.execute(MySqlSnapshotSplitReadTask.java:115) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
... 6 more
2022-09-13 11:24:36,860 ERROR org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager [] - Received uncaught exception.
java.lang.RuntimeException: SplitFetcher thread 5317 received unexpected exception while polling the records
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:150) ~[flink-table_2.11-1.14.5.jar:1.14.5]
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:105) [flink-table_2.11-1.14.5.jar:1.14.5]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_342]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_342]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_342]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_342]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_342]
Caused by: org.apache.flink.util.FlinkRuntimeException: Read split MySqlSnapshotSplit{tableId=gc_fps_receivable.headorderdetail, splitId='gc_fps_receivable.headorderdetail:0', splitKeyType=[`HeadOrderDetailId` INT NOT NULL], splitStart=null, splitEnd=null, highWatermark=null} error due to java.lang.NullPointerException.
at com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader.checkReadException(SnapshotSplitReader.java:256) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
at com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader.pollSplitRecords(SnapshotSplitReader.java:233) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
at com.ververica.cdc.connectors.mysql.source.reader.MySqlSplitReader.fetch(MySqlSplitReader.java:78) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
at org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58) ~[flink-table_2.11-1.14.5.jar:1.14.5]
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:142) ~[flink-table_2.11-1.14.5.jar:1.14.5]
... 6 more
Caused by: io.debezium.DebeziumException: java.lang.NullPointerException
at com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.execute(MySqlSnapshotSplitReadTask.java:120) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
at com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader.lambda$submitSplit$0(SnapshotSplitReader.java:117) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
... 5 more
Caused by: java.lang.NullPointerException
at io.debezium.util.ColumnUtils.toArray(ColumnUtils.java:41) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
at com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.createDataEventsForTable(MySqlSnapshotSplitReadTask.java:230) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
at com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.createDataEvents(MySqlSnapshotSplitReadTask.java:191) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
at com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.doExecute(MySqlSnapshotSplitReadTask.java:150) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
at com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.execute(MySqlSnapshotSplitReadTask.java:115) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
at com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader.lambda$submitSplit$0(SnapshotSplitReader.java:117) ~[blob_p-f6c45d5acd537a762ce2c8c3ba78cd6f86d3aa19-1fa09c9d840fdb769f95aa5c78d807aa:?]
... 5 more
2022-09-13 11:24:36,862 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSplitReader [] - Close current debezium reader com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader
2022-09-13 11:24:36,863 INFO org.apache.flink.connector.base.source.reader.SourceReaderBase [] - Closing Source Reader.
2022-09-13 11:24:36,863 INFO org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher [] - Shutting down split fetcher 5317
2022-09-13 11:24:36,864 INFO io.debezium.jdbc.JdbcConnection [] - Connection gracefully closed
2022-09-13 11:24:36,865 INFO org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher [] - Split fetcher 5317 exited.
2022-09-13 11:24:36,867 WARN org.apache.flink.runtime.taskmanager.Task [] - Source: MySQL Source (1/1)#0 (3daab7b1bd82a73f73ab67bd389c51c3) switched from RUNNING to FAILED with failure cause: java.lang.RuntimeException: One or more fetchers have encountered exception
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:225)
at org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169)
at org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:130)
at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:354)
at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:496)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: SplitFetcher thread 5317 received unexpected exception while polling the records
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:150)
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:105)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.flink.util.FlinkRuntimeException: Read split MySqlSnapshotSplit{tableId=gc_fps_receivable.headorderdetail, splitId='gc_fps_receivable.headorderdetail:0', splitKeyType=[`HeadOrderDetailId` INT NOT NULL], splitStart=null, splitEnd=null, highWatermark=null} error due to java.lang.NullPointerException.
at com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader.checkReadException(SnapshotSplitReader.java:256)
at com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader.pollSplitRecords(SnapshotSplitReader.java:233)
at com.ververica.cdc.connectors.mysql.source.reader.MySqlSplitReader.fetch(MySqlSplitReader.java:78)
at org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58)
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:142)
... 6 more
Caused by: io.debezium.DebeziumException: java.lang.NullPointerException
at com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.execute(MySqlSnapshotSplitReadTask.java:120)
at com.ververica.cdc.connectors.mysql.debezium.reader.SnapshotSplitReader.lambda$submitSplit$0(SnapshotSplitReader.java:117)
... 5 more
Caused by: java.lang.NullPointerException
at io.debezium.util.ColumnUtils.toArray(ColumnUtils.java:41)
at com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.createDataEventsForTable(MySqlSnapshotSplitReadTask.java:230)
at com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.createDataEvents(MySqlSnapshotSplitReadTask.java:191)
at com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.doExecute(MySqlSnapshotSplitReadTask.java:150)
at com.ververica.cdc.connectors.mysql.debezium.task.MySqlSnapshotSplitReadTask.execute(MySqlSnapshotSplitReadTask.java:115)
... 6 more
I've had this problem several times .Occurs when a table structure is modified during snapshotting
一模一样的问题,我的库名是带小数点的,感觉是库名问题,但是加上'scan.startup.mode' = 'latest-offset',又没问题了,感觉是binlog日志格式问题,具体原因为未找到,楼主怎么解决的
一模一样的问题,我的库名是带小数点的,感觉是库名问题,但是加上'scan.startup.mode' = 'latest-offset',又没问题了,感觉是binlog日志格式问题,具体原因为未找到,楼主怎么解决的
也遇到了库名带小数点的
