binlog2sql UnicodeDecodeError: 'utf-8' codec can't decode byte xxx

python binlog2sql/binlog2sql.py xxxxxxxxxxx --start-file='mysqld.000006' --start-datetime='2018-06-24 13:30:00' --start-position=170802294 -B >callback_script.sql 报错：（代码已更新至目前——20180624 14:16时的master版本） Traceback (most recent call last): File "binlog2sql/binlog2sql.py", line 150, in binlog2sql.process_binlog() File "binlog2sql/binlog2sql.py", line 121, in process_binlog self.print_rollback_sql(filename=tmp_file) File "binlog2sql/binlog2sql.py", line 129, in print_rollback_sql for line in reversed_lines(f_tmp): File "/root/softwares/binlog2sql/binlog2sql/binlog2sql_util.py", line 243, in reversed_lines block = block.decode("utf-8") UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbc in position 0: invalid start byte

不加-B的话，能够正常解析出错误的操作，但是加了-B就报上面的错误，内容是中文，有一些有全角或半角的中文符号

Jun 24 '18 06:06 ericwtlin

Jul 03 '18 09:07 weathery

请用mysqlbinlog解析出报错那行数据，看下是否有乱码。这种情况一般是，连接是gbk，表结构是utf8，实际存的是gbk编码的中文。用utf8去解码时，mysql会解出乱码，python decode()会抛出UnicodeDecodeError

Jul 23 '18 03:07 danfengcao

报差不多的错.求解决.

 Traceback (most recent call last):
  File "binlog2sql.py", line 150, in <module>
    binlog2sql.process_binlog()
  File "binlog2sql.py", line 74, in process_binlog
    for binlog_event in stream:
  File "/usr/lib/python2.7/site-packages/pymysqlreplication/binlogstream.py", line 432, in fetchone
    self.__fail_on_table_metadata_unavailable)
  File "/usr/lib/python2.7/site-packages/pymysqlreplication/packet.py", line 139, in __init__
    fail_on_table_metadata_unavailable=fail_on_table_metadata_unavailable)
  File "/usr/lib/python2.7/site-packages/pymysqlreplication/event.py", line 175, in __init__
    - self.schema_length - 1).decode("utf-8")
  File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xcf in position 417: invalid continuation byte

Aug 07 '18 06:08 Jamy325

也卡在这个bugs

Oct 10 '18 10:10 xiaopenggithub

也开在这个bug,请问有什么解决方法吗

Oct 28 '18 09:10 zgmqdl

将binlog2sql_util.py中的decode("utf-8")替换为decode("utf-8","ignore")

Nov 06 '18 03:11 dy7338

也开在这个bug,请问有什么解决方法吗

请看我的回复

Nov 06 '18 03:11 dy7338

Windows平台通过GBK解码即可：

import platform
...
def reversed_lines(fin):
    """Generate the lines of file in reverse order."""
    part = ''
    for block in reversed_blocks(fin):
        if PY3PLUS:
            # block =  block.decode("utf-8")
            block = platform.system() == 'Windows' and block.decode("gbk") or block.decode("utf-8")
        for c in reversed(block):
            if c == '\n' and part:
                yield part[::-1]
                part = ''
            part += c
    if part:
        yield part[::-1]

@danfengcao 你好，这个报错一直存在，如果有时间的话，请修复一下~