mysql-binlog-connector-java icon indicating copy to clipboard operation
mysql-binlog-connector-java copied to clipboard

Issue with date transformation using debezium

Open guilhermeassad opened this issue 4 years ago • 0 comments

We use Debezium with MySQL connector to get data from a MariaDB server. A few months ago we upgraded our MariaDB to version 10.4 and after that, we started having this issue with the connectors. Some colummns on a table are datetime(6).

Task threw an uncaught and unrecoverable exception
org.apache.kafka.connect.errors.ConnectException: Invalid value for MonthOfYear (valid values 1 - 12): -83
        at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:241)
        at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:207)
        at io.debezium.connector.mysql.BinlogReader$ReaderThreadLifecycleListener.onEventDeserializationFailure(BinlogReader.java:1150)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:965)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:599)
        at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:857)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.time.DateTimeException: Invalid value for MonthOfYear (valid values 1 - 12): -83
        at java.base/java.time.temporal.ValueRange.checkValidValue(ValueRange.java:311)
        at java.base/java.time.temporal.ChronoField.checkValidValue(ChronoField.java:717)
        at java.base/java.time.LocalDate.of(LocalDate.java:269)
        at java.base/java.time.LocalDateTime.of(LocalDateTime.java:361)
        at io.debezium.connector.mysql.RowDeserializers.deserializeDatetime(RowDeserializers.java:378)
        at io.debezium.connector.mysql.RowDeserializers$WriteRowsDeserializer.deserializeDatetime(RowDeserializers.java:202)
        at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeCell(AbstractRowsEventDataDeserializer.java:181)
        at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeRow(AbstractRowsEventDataDeserializer.java:143)
        at com.github.shyiko.mysql.binlog.event.deserialization.WriteRowsEventDataDeserializer.deserializeRows(WriteRowsEventDataDeserializer.java:64)
        at com.github.shyiko.mysql.binlog.event.deserialization.WriteRowsEventDataDeserializer.deserialize(WriteRowsEventDataDeserializer.java:56)
        at com.github.shyiko.mysql.binlog.event.deserialization.WriteRowsEventDataDeserializer.deserialize(WriteRowsEventDataDeserializer.java:32)
        at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:303)
        at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:232)
        at io.debezium.connector.mysql.BinlogReader$1.nextEvent(BinlogReader.java:252)
        at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:952)
        ... 3 more

We noticed from the binlog file that although the statement is correct the fields have a strange date format.

#Q> INSERT INTO `some_table` (`tsCreated`, `tsUpdated`, `label_id`, `customer_id`, `category_id`, `product_id`, `te_id`, `lot_id`, `page`) VALUES ('2021-11-01 10:46:25.219851', '2021-11-01 10:46:25.219878'
, 1, NULL, NULL, 139153, '017c8322f53e00059e3b6c9d48bb000780021070004a0', NULL, 'auction')
#211101 11:46:25 server id 204203  end_log_pos 170739039 CRC32 0x055ccef1       Table_map: `database`.`some_table` mapped to number 97
# at 170739039
#211101 11:46:25 server id 204203  end_log_pos 170739157 CRC32 0xc82864ea       Write_rows: table id 97 flags: STMT_END_F
### INSERT INTO `database`.`some_table`
### SET
###   @1=219190348
###   @2='85489092-40-24 33:28:01'
###   @3='280044596-30-48 38:70:73'
###   @4=1
###   @5='017c8322f53e00059e3b6c9d48bb000780021070004a0'
###   @6='auction'
###   @7=139153
###   @8=NULL
###   @9=NULL
###   @10=NULL
# Number of rows: 1
# at 170739157
#211101 11:46:25 server id 204203  end_log_pos 170739188 CRC32 0x8dec9c7a       Xid = 2376154080
COMMIT/*!*/;

This binlog is already from a replicated database, but the master shows the same behavior, and looking into the database itself, the data is all correct. So, we believe that problem is not in MariaDB itself but in the client-side. And the issue is pretty similar to this one: https://jira.mariadb.org/browse/MDEV-20860 We did try to upgrade java, Debezium, and our latest try was to upgrade only this binlog library to version 0.25.4, but nothing helped.

guilhermeassad avatar Nov 03 '21 12:11 guilhermeassad