Could not parse the binlog of Mysql >= 8.0.23

[tinntsea]

Mysql has added a new feature called visible columns in version 8.0.23 (but we have not used it).

https://dev.mysql.com/doc/dev/mysql-server/8.0.21/classbinary__log_1_1Table__map__event.html#a1b84e5b226c76eaf9c0df8ed03ba1393

https://dev.mysql.com/doc/dev/mysql-server/8.0.23/classbinary__log_1_1Table__map__event.html#a1b84e5b226c76eaf9c0df8ed03ba1393

https://dev.mysql.com/doc/refman/8.0/en/invisible-columns.html

The binlog connector does not handle the new value COLUMN_VISIBILITY (12) in TableMapEventMetadata: https://github.com/osheroff/mysql-binlog-connector-java/blob/0.25.1/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/TableMapEventMetadataDeserializer.java#L81

https://github.com/osheroff/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/TableMapEventMetadataDeserializer.java#L184 will return null, which triggers NullPointerException at it caller at https://github.com/osheroff/mysql-binlog-connector-java/blob/master/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/TableMapEventMetadataDeserializer.java#L47

[gunnarmorling]

Hey @osheroff, I was thinking about looking into this one; struggling a bit though with the test infrastructure; when trying to spin up 8.0.23 with that onetimeserver utility, it's always failing:

src/test/onetimeserver --mysql-version=8.0.23 --log-slave-updates --log-bin=master --binlog_format=row --innodb_flush_log_at_trx_commit=0 --server_id=1 --character-set-server=utf8 --sync_binlog=0 --default-time-zone=+00:00 --gtid-mode=ON --log-slave-updates=ON --enforce-gtid-consistency=true
Child exited without printing info!

Same when trying 8.0 or 8.0.12 (as present here). Any idea what could be the reason?

[osheroff]

so yeah, I need to extract or roll mysqld binaries for these platforms. what OS are you running?

[osheroff]

8.0.25 is up and running on OSX (after you do "onetimeserver update" inside mysql-binlog-connector)

[gunnarmorling]

Hey @osheroff, thx! I believe the onetimeserver-go binary here still needs updating though, correct? Otherwise, the change you did to mysql.go for adding the new version won't be applied. If I build the binary locally, I can use 8.0.25 successfully.

[gunnarmorling]

Hey @tinntsea, so I've created a PR which should fix this (see https://github.com/osheroff/mysql-binlog-connector-java/pull/41), but I'm struggling to reproduce the issue to begin with. Whatever I do, I'll never receive that new metadata type for INVISIBILITY.

Can you perhaps share the sequence of DDL + DML which was triggering the issue for you in Debezium? I then could update the test case in the PR accordingly. Thanks!

[tinntsea]

@gunnarmorling (I replied in your PR as well)

We need to set --binlog-row-metadata=FULL in mysql configuration in order to get the invisible metadata.

https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_row_metadata