Json null not handled correctly

[rahilsh]

We are using pg_chameleon for replicating data from MySQL to PostgreSQL and pg_chameleon is using python-MySQL-replication internally. We are getting below error during replication. Is there any way to resolve it

2018-09-04 13:13:49 MainProcess INFO: Starting the replica daemons for source mysql 
2018-09-04 18:32:21 MainProcess ERROR: Read process alive: False - Replay process alive: True
2018-09-04 18:32:21 MainProcess ERROR: Stack trace: Traceback (most recent call last):
  File "/var/log/bucardo/venv/lib64/python3.6/site-packages/pg_chameleon/lib/global_lib.py", line 487, in read_replica
    self.mysql_source.read_replica()
  File "/var/log/bucardo/venv/lib64/python3.6/site-packages/pg_chameleon/lib/mysql_lib.py", line 1336, in read_replica
    replica_data=self.__read_replica_stream(batch_data)
  File "/var/log/bucardo/venv/lib64/python3.6/site-packages/pg_chameleon/lib/mysql_lib.py", line 1199, in __read_replica_stream
    for row in binlogevent.rows:
  File "/var/log/bucardo/venv/lib64/python3.6/site-packages/pymysqlreplication/row_event.py", line 429, in rows
    self._fetch_rows()
  File "/var/log/bucardo/venv/lib64/python3.6/site-packages/pymysqlreplication/row_event.py", line 424, in _fetch_rows
    self.__rows.append(self._fetch_one_row())
  File "/var/log/bucardo/venv/lib64/python3.6/site-packages/pymysqlreplication/row_event.py", line 513, in _fetch_one_row
    row["before_values"] = self._read_column_data(self.columns_present_bitmap)
  File "/var/log/bucardo/venv/lib64/python3.6/site-packages/pymysqlreplication/row_event.py", line 181, in _read_column_data
    values[name] = self.packet.read_binary_json(column.length_size)
  File "/var/log/bucardo/venv/lib64/python3.6/site-packages/pymysqlreplication/packet.py", line 352, in read_binary_json
    return self.read_binary_json_type(t, length)
  File "/var/log/bucardo/venv/lib64/python3.6/site-packages/pymysqlreplication/packet.py", line 357, in read_binary_json_type
    return self.read_binary_json_object(length - 1, large)
  File "/var/log/bucardo/venv/lib64/python3.6/site-packages/pymysqlreplication/packet.py", line 416, in read_binary_json_object
    raise ValueError('Json length is larger than packet length')
ValueError: Json length is larger than packet length

2018-09-04 18:32:21 MainProcess ERROR: Read daemon crashed. Terminating the replay daemon.
2018-09-04 18:32:21 MainProcess INFO: Replica process for source MySQL ended

[baloo]

Any chance you could extract the data that fails to parse? Something like this would do:

diff --git a/pymysqlreplication/packet.py b/pymysqlreplication/packet.py
index 936dc8c5b39d5..53edb6e18d7fc 100644
--- a/pymysqlreplication/packet.py
+++ b/pymysqlreplication/packet.py
@@ -1,6 +1,7 @@
 # -*- coding: utf-8 -*-
 
 import struct
+import binascii
 
 from pymysql.util import byte2int
 
@@ -349,7 +349,12 @@ class BinLogPacketWrapper(object):
         self.unread(payload)
         t = self.read_uint8()
 
-        return self.read_binary_json_type(t, length)
+        try:
+            return self.read_binary_json_type(t, length)
+        except:
+            print(length)
+            print(binascii.hexlify(payload))
+            raise
 
     def read_binary_json_type(self, t, length):
         large = (t in (JSONB_TYPE_LARGE_OBJECT, JSONB_TYPE_LARGE_ARRAY))

If you can't give data here (because corporate/customer data), could you please contribute a test that fails? That would help a lot debugging your issue. Thank you!

[rahilsh]

@baloo Sure will try that. Meanwhile, I tried changing JSON to JSONB and issue is not happening now.

[rahilsh]

@baloo I see this on terminal

0
b''

[rahilsh]

For further debugging, I printed column name and figured out the value in DB and it turned out to be a JSON null. Any way to fix this?

[the4thdoctor]

I'm trying to reproduce the issue on pg_chameleon, however the replica works fine when I run insert into test_json (field_json) values (cast('null' AS JSON));

my configuration is mysql 5.7.23 on freebsd 11.2, python 3.6.

could it be a not conventional packet coming out from RDS?

[rahilsh]

I also printed column name, type and column.length_size using print("colume:"+str(column.name)+". type: "+str(column.type)+". lsize: "+str(column.length_size)) just before line 181 of row_event.py and it printed colume:status_dates. type: 245. lsize: 4

[rahilsh]

@baloo Any idea who can help.

[rudaporto]

Any update on this? I've got the same error:

Traceback (most recent call last):

  File "/srv/lazarus/src/lazarus/mysql_reader.py", line 306, in run
    for row in binlogevent.rows:

  File "/usr/local/src/pypy3.6/site-packages/pymysqlreplication/row_event.py", line 429, in rows
    self._fetch_rows()

  File "/usr/local/src/pypy3.6/site-packages/pymysqlreplication/row_event.py", line 424, in _fetch_rows
    self.__rows.append(self._fetch_one_row())

  File "/usr/local/src/pypy3.6/site-packages/pymysqlreplication/row_event.py", line 513, in _fetch_one_row
    row["before_values"] = self._read_column_data(self.columns_present_bitmap)

  File "/usr/local/src/pypy3.6/site-packages/pymysqlreplication/row_event.py", line 181, in _read_column_data
    values[name] = self.packet.read_binary_json(column.length_size)

  File "/usr/local/src/pypy3.6/site-packages/pymysqlreplication/packet.py", line 352, in read_binary_json
    return self.read_binary_json_type(t, length)

  File "/usr/local/src/pypy3.6/site-packages/pymysqlreplication/packet.py", line 357, in read_binary_json_type
    return self.read_binary_json_object(length - 1, large)

  File "/usr/local/src/pypy3.6/site-packages/pymysqlreplication/packet.py", line 416, in read_binary_json_object
    raise ValueError('Json length is larger than packet length')

ValueError: Json length is larger than packet length