server icon indicating copy to clipboard operation
server copied to clipboard

MDEV-32570: Fragment ROW replication events larger than slave_max_allowed_packet

Open bnestere opened this issue 6 months ago • 0 comments
trafficstars

This PR is currently a draft.

This PR solves two problems:

  1. Rows log events cannot be transmitted to the slave if their size exceeds slave_max_packet_size (max 1GB at the time of writing this patch, i.e. MariaDB 12.1)
  2. Rows log events cannot be binlogged if they are larger than 4GB because the common binlog event header field event_len is 32-bits.

This PR adds support for fragmenting large Rows_log_events through a new event type, Partial_rows_log_event. When any given instantiation of a Rows_log_event (e.g. Write_rows_log_event, etc) is too large to be sent to a replica (i.e. larger than the value slave_max_allowed_packet, as configured on a replica), then the rows event must be fragmented into sub-events (i.e. Partial_rows_log_events), each of size slave_max_allowed_packet, so the event can be transmitted to the replica. The replica will then take the content of each of these Partial_rows_log_events, and join them together into a large Rows_log_event to be executed as normal. Partial_rows_log_events are written to the binary log sequentially, and the replica assembles the events in the order they are binlogged.

Remaining things to be done to remove draft status of the PR:

  1. Implement error handling
  2. Implementing Partial_rows_log_event::pack_info()
  3. Extend MTR testing

This PR is organized as follows:

  • Commits 1 - 3 commits add code preparations to make the actual feature commits easier to read
  • Commit 4 adds the Partial_rows_log_event type and the server logic to support fragmenting and re-assembling/applying large Rows_log_events.
  • Commit 5 adds client (mysqlbinlog) logic to support output and replay of Partial_rows_log_events through piping to the mariadb client
  • Commit 6 adds the MTR test (still working on this) for the feature
  • Commit 7 adjusts existing MTR tests to pass

Note that the git commit messages provide much more specific details on the implementation.

bnestere avatar May 16 '25 21:05 bnestere