Mariadb 10.6.12: Mitigate corrupt binlog event bug (#399)

After upgrading our production databases from MariaDB 10.4 to 10.6.12, we started running into an issue which appears to be a bug in this Mariadb version, see the stacktrace:

Traceback (most recent call last):
  File "/projects/python-mysql-replication/examples/mariadb_gtid/read_event.py", line 80, in <module>
    for binlogevent in stream:
  File "/projects/python-mysql-replication/pymysqlreplication/binlogstream.py", line 490, in fetchone
    pkt = self._stream_connection._read_packet()
  File "/projects/python-mysql-replication/venv/lib/python3.8/site-packages/pymysql/connections.py", line 729, in _read_packet
    packet.raise_for_error()
  File "/projects/python-mysql-replication/venv/lib/python3.8/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/projects/PycharmProjects/python-mysql-replication/venv/lib/python3.8/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1236, "Failed to replace GTID event with backwards-compatible event: corrupt event.; the first event 'mysql-bin-changelog.310754' at 18888372, the last event read from 'mysql-bin-changelog.310754' at 18888372, the last byte read from 'mysql-bin-changelog.310754' at 18888418.")

This error is coming from this function in Mariadb's source code.

The bug is triggered when using BinLogStreamReader with binlog coordinates (not GTID), the code doesn't specify the slave capability thus the condition mariadb_slave_capability < MARIA_SLAVE_CAPABILITY_GTID passes and Mariadb then tries to replace the GTID event with a dummy one but fails, we can mitigate this by setting mariadb_slave_capability=4 in the non-gtid path too (as it's done in the gtid one), it would not be an issue because the library has had support for MariaDB GTID events for sometime now.

This fix has been tested against the same production DB and it's working fine and we receive a MariaDBGtidEvent:

=== MariadbGtidEvent ===
Date: 2023-04-10T03:19:03
Log position: 18888418
Event size: 23
Read bytes: 13
Flags: 14
GTID: 0-201709275-3720522848

Running the query set @mariadb_slave_capability=4 against mysql doesn't do anything hence no need I reckon to run this in an if block which would require knowing the DB engine (mysql or mariadb), there is a flag is_mariadb but it's false and not required for the non-gtid path, using it would make this a breaking change for downstream users so I'm trying to avoid that, but needs be, I can make it dynamic to detect the DB engine type.
teaf-wise reacted with thumbs up emoji

Author: Samira-El

pymysqlreplication/binlogstream.py

@@ -319,6 +319,14 @@ def __connect_to_stream(self):
             cur.execute("set @master_heartbeat_period= %d" % heartbeat)
             cur.close()
 
+        # When replicating from Mariadb 10.6.12 using binlog coordinates, a slave capability < 4 triggers a bug in
+        # Mariadb, when it tries to replace GTID events with dummy ones. Given that this library understands GTID
+        # events, setting the capability to 4 circumvents this error.
+        # If the DB is mysql, this won't have any effect so no need to run this in a condition
+        cur = self._stream_connection.cursor()
+        cur.execute("SET @mariadb_slave_capability=4")
+        cur.close()
+
         self._register_slave()
 
         if not self.auto_position:
@@ -352,8 +360,6 @@ def __connect_to_stream(self):
             if self.is_mariadb:
                 # https://mariadb.com/kb/en/5-slave-registration/
                 cur = self._stream_connection.cursor()
-
-                cur.execute("SET @mariadb_slave_capability=4")
                 cur.execute("SET @slave_connect_state='%s'" % self.auto_position)
                 cur.execute("SET @slave_gtid_strict_mode=1")
                 cur.execute("SET @slave_gtid_ignore_duplicates=0")