Received message could not be parsed

The problem

From time to time the following error occurs:

2017-01-11 17:41:31,926 Received Message: b'{"unread": false, "event": "read", "id": "01000000a6b4ce0fae0000000000000077616c38a95a2f29", "flags": 256, "out": false, "from": {"id": "$01000000a6b4ce0f77616c38a95a2f29", "peer_type": "user", "peer_id": 265204902, "print_name": "Chat_Wars", "first_name": "Chat Wars", "flags": 1, "last_name": "", "username": "ChatWarsBot"}, "to": {"id": "$010000009fb8ba120b62fa795ec02cf3", "peer_type": "user", "peer_id": 314226847, "when": "2017-01-11 17:42:11", "print_name": "Mike_Udkovsky", "flags": 524289, "first_name": "Mike", "phone": "79671494739", "last_name": "Udkovsky", "username": "mikeud"}, "service": false, "date": 1484127604, "text": "\xd0\xa2\xd1\x8b \xd0\xbf\xd1\x80\xd0\xb8\xd0\xb3\xd0\xbe\xd1\x82\xd0\xbe\xd0\xb2\xd0\xb8\xd0\xbb\xd1\x81\xd1\x8f \xd0\xba \xd0\xb7\xd0\xb0\xd1\x89\xd0\xb8\xd1\x82\xd0\xb5. \xd0\x91\xd0\xbb\xd0\xb8\xd0\xb6\xd0\xb0\xd0\xb9\xd1\x88\xd0\xb5\xd0\xb5 \xd1\x81\xd1\x80\xd0\xb0\xd0\xb6\xd0\xb5\xd0\xbd\xd0\xb8\xd0\xb5 \xd1\x87\xd0\xb5\xd1\x80\xd0\xb5\xd0\xb7 20\xd0\xbc\xd0\xb8\xd0\xbd\xd1\x83\xd1\x82. \xd0\x96\xd0\xb4\xd0\xb8 \xd1\x80\xd0\xb5\xd0\xb7\xd1\x83\xd0\xbb\xd1\x8c\xd1\x82\xd0\xb0\xd1\x82\xd0\xb0."}\n{"unread": false, "event": "read", "id": "02000000b110f70aac000000000000000000000000000000", "flags": 256, "out": false, "reply_id": "02000000b110f70aa4000000000000000000000000000000", "to": {"id": "$02000000b110f70a0000000000000000", "peer_type": "chat", "peer_id": 183963825, "print_name": "\xd0\x9c\xd0\xb0\xd1\x81\xd0\xbe\xd0\xbd\xd1\x81\xd0\xba\xd0\xb0\xd1\x8f_\xd0\xbb\xd0\xbe\xd0\xb6\xd0\xb0_\xf0\x9f\x87\xa8\xf0\x9f\x87\xbe_\xf0\x9f\x87\xae\xf0\x9f\x87\xb2_\xf0\x9f\x87\xbb\xf0\x9f\x87\xa6_\xf0\x9f\x87\xac\xf0\x9f\x87\xb5_\xf0\x9f\x87\xaa\xf0\x9f\x87\xba", "flags": 65537, "title": "\xd0\x9c\xd0\xb0\xd1\x81\xd0\xbe\xd0\xbd\xd1\x81\xd0\xba\xd0\xb0\xd1\x8f \xd0\xbb\xd0\xbe\xd0\xb6\xd0\xb0 \xf0\x9f\x87\xa8\xf0\x9f\x87\xbe \xf0\x9f\x87\xae\xf0\x9f\x87\xb2 \xf0\x9f\x87\xbb\xf0\x9f\x87\xa6 \xf0\x9f\x87\xac\xf0\x9f\x87\xb5 \xf0\x9f\x87\xaa\xf0\x9f\x87\xba", "admin": {"id": "$01000000000000000000000000000000", "peer_type": "user", "peer_id": 0, "print_name": "user#0"}, "members_num": 5}, "from": {"id": "$010000005ac62f129716439697cfeaf8", "peer_type": "user", "peer_id": 305120858, "when": "2017-01-11 11:44:32", "print_name": "Octo_Cat", "flags": 1, "first_name": "Octo", "last_name": "Cat", "username": "octo_cat"}, "service": false, "date": 1484127448, "text": "\xd0\x9f\xd0\xbe\xd0\xbd\xd1\x8f\xd1\x82\xd0\xbd\xd0\xbe, \xd1\x81\xd1\x82\xd0\xb0\xd0\xb2\xd0\xbb\xd1\x8e: \xe2\x9a\x94 \xd0\x90\xd1\x82\xd0\xb0\xd0\xba\xd0\xb0, \xf0\x9f\x87\xac\xf0\x9f\x87\xb5"}'
2017-01-11 17:41:31,927 Received Message: "{"unread": false, "event": "read", "id": "01000000a6b4ce0fae0000000000000077616c38a95a2f29", "flags": 256, "out": false, "from": {"id": "$01000000a6b4ce0f77616c38a95a2f29", "peer_type": "user", "peer_id": 265204902, "print_name": "Chat_Wars", "first_name": "Chat Wars", "flags": 1, "last_name": "", "username": "ChatWarsBot"}, "to": {"id": "$010000009fb8ba120b62fa795ec02cf3", "peer_type": "user", "peer_id": 314226847, "when": "2017-01-11 17:42:11", "print_name": "Mike_Udkovsky", "flags": 524289, "first_name": "Mike", "phone": "79671494739", "last_name": "Udkovsky", "username": "mikeud"}, "service": false, "date": 1484127604, "text": "Ты приготовился к защите. Ближайшее сражение через 20минут. Жди результата."}
{"unread": false, "event": "read", "id": "02000000b110f70aac000000000000000000000000000000", "flags": 256, "out": false, "reply_id": "02000000b110f70aa4000000000000000000000000000000", "to": {"id": "$02000000b110f70a0000000000000000", "peer_type": "chat", "peer_id": 183963825, "print_name": "Масонская_ложа_🇨🇾_🇮🇲_🇻🇦_🇬🇵_🇪🇺", "flags": 65537, "title": "Масонская ложа 🇨🇾 🇮🇲 🇻🇦 🇬🇵 🇪🇺", "admin": {"id": "$01000000000000000000000000000000", "peer_type": "user", "peer_id": 0, "print_name": "user#0"}, "members_num": 5}, "from": {"id": "$010000005ac62f129716439697cfeaf8", "peer_type": "user", "peer_id": 305120858, "when": "2017-01-11 11:44:32", "print_name": "Octo_Cat", "flags": 1, "first_name": "Octo", "last_name": "Cat", "username": "octo_cat"}, "service": false, "date": 1484127448, "text": "Понятно, ставлю: ⚔ Атака, 🇬🇵"}"
2017-01-11 17:41:31,930 Received message could not be parsed.
Message:>{"unread": false, "event": "read", "id": "01000000a6b4ce0fae0000000000000077616c38a95a2f29", "flags": 256, "out": false, "from": {"id": "$01000000a6b4ce0f77616c38a95a2f29", "peer_type": "user", "peer_id": 265204902, "print_name": "Chat_Wars", "first_name": "Chat Wars", "flags": 1, "last_name": "", "username": "ChatWarsBot"}, "to": {"id": "$010000009fb8ba120b62fa795ec02cf3", "peer_type": "user", "peer_id": 314226847, "when": "2017-01-11 17:42:11", "print_name": "Mike_Udkovsky", "flags": 524289, "first_name": "Mike", "phone": "79671494739", "last_name": "Udkovsky", "username": "mikeud"}, "service": false, "date": 1484127604, "text": "Ты приготовился к защите. Ближайшее сражение через 20минут. Жди результата."}
{"unread": false, "event": "read", "id": "02000000b110f70aac000000000000000000000000000000", "flags": 256, "out": false, "reply_id": "02000000b110f70aa4000000000000000000000000000000", "to": {"id": "$02000000b110f70a0000000000000000", "peer_type": "chat", "peer_id": 183963825, "print_name": "Масонская_ложа_🇨🇾_🇮🇲_🇻🇦_🇬🇵_🇪🇺", "flags": 65537, "title": "Масонская ложа 🇨🇾 🇮🇲 🇻🇦 🇬🇵 🇪🇺", "admin": {"id": "$01000000000000000000000000000000", "peer_type": "user", "peer_id": 0, "print_name": "user#0"}, "members_num": 5}, "from": {"id": "$010000005ac62f129716439697cfeaf8", "peer_type": "user", "peer_id": 305120858, "when": "2017-01-11 11:44:32", "print_name": "Octo_Cat", "flags": 1, "first_name": "Octo", "last_name": "Cat", "username": "octo_cat"}, "service": false, "date": 1484127448, "text": "Понятно, ставлю: ⚔ Атака, 🇬🇵"}<
Traceback (most recent call last):
  File "C:\Users\Nonamed\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pytg\receiver.py", line 182, in _add_message
    json_dict = json.loads(text)
  File "C:\Users\Nonamed\AppData\Local\Programs\Python\Python36-32\lib\json\__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "C:\Users\Nonamed\AppData\Local\Programs\Python\Python36-32\lib\json\decoder.py", line 342, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 718)

2017-01-11 20:27:05,291 Received Message: b'{"event": "read", "reply_id": "02000000b110f70ac6000000000000000000000000000000", "id": "02000000b110f70acd000000000000000000000000000000", "flags": 256, "out": false, "from": {"peer_id": 314998971, "id": "$01000000bb80c61256f8c2ffc2450486", "when": "2017-01-11 20:04:52", "peer_type": "user", "print_name": "\xd0\x9f\xd0\x9e\xd0\x9a\xd0\xa0\xd0\x90\xd0\xa1_\xd0\x9e\xd0\x91\xd0\x9e\xd0\x95\xd0\x92", "username": "pokras_oboev", "flags": 1, "first_name": "\xd0\x9f\xd0\x9e\xd0\x9a\xd0\xa0\xd0\x90\xd0\xa1", "last_name": "\xd0\x9e\xd0\x91\xd0\x9e\xd0\x95\xd0\x92"}, "to": {"peer_id": 183963825, "id": "$02000000b110f70a0000000000000000", "peer_type": "chat", "title": "\xd0\x9c\xd0\xb0\xd1\x81\xd0\xbe\xd0\xbd\xd1\x81\xd0\xba\xd0\xb0\xd1\x8f \xd0\xbb\xd0\xbe\xd0\xb6\xd0\xb0 \xf0\x9f\x87\xa8\xf0\x9f\x87\xbe \xf0\x9f\x87\xae\xf0\x9f\x87\xb2 \xf0\x9f\x87\xbb\xf0\x9f\x87\xa6 \xf0\x9f\x87\xac\xf0\x9f\x87\xb5 \xf0\x9f\x87\xaa\xf0\x9f\x87\xba", "print_name": "\xd0\x9c\xd0\xb0\xd1\x81\xd0\xbe\xd0\xbd\xd1\x81\xd0\xba\xd0\xb0\xd1\x8f_\xd0\xbb\xd0\xbe\xd0\xb6\xd0\xb0_\xf0\x9f\x87\xa8\xf0\x9f\x87\xbe_\xf0\x9f\x87\xae\xf0\x9f\x87\xb2_\xf0\x9f\x87\xbb\xf0\x9f\x87\xa6_\xf0\x9f\x87\xac\xf0\x9f\x87\xb5_\xf0\x9f\x87\xaa\xf0\x9f\x87\xba", "admin": {"peer_id": 0, "id": "$01000000000000000000000000000000", "peer_type": "user", "print_name": "user#0"}, "flags": 65537, "members_num": 5}, "unread": false, "text": "\xd0\xbb\xd0\xb0\xd0\xb4\xd0\xbd\xd0\xbe: \xe2\x9a\x94 \xd0\x90\xd1\x82\xd0\xb0\xd0\xba\xd0\xb0, \xf0\x9f\x87\xac\xf0\x9f\x87\xb5", "date": 1484136045, "service": false}\n{"event": "read", "id": "01000000a6b4ce0fca0000000000000077616c38a95a2f29", "flags": 256, "out": false, "text": "\xd0\xa2\xd1\x8b \xd0\xbf\xd1\x80\xd0\xb8\xd0\xb3\xd0\xbe\xd1\x82\xd0\xbe\xd0\xb2\xd0\xb8\xd0\xbb\xd1\x81\xd1\x8f \xd0\xba \xd0\xb0\xd1\x82\xd0\xb0\xd0\xba\xd0\xb5. \xd0\x91\xd0\xbb\xd0\xb8\xd0\xb6\xd0\xb0\xd0\xb9\xd1\x88\xd0\xb5\xd0\xb5 \xd1\x81\xd1\x80\xd0\xb0\xd0\xb6\xd0\xb5\xd0\xbd\xd0\xb8\xd0\xb5 \xd1\x87\xd0\xb5\xd1\x80\xd0\xb5\xd0\xb7 59\xd0\xbc\xd0\xb8\xd0\xbd\xd1\x83\xd1\x82. \xd0\x96\xd0\xb4\xd0\xb8 \xd1\x80\xd0\xb5\xd0\xb7\xd1\x83\xd0\xbb\xd1\x8c\xd1\x82\xd0\xb0\xd1\x82\xd0\xb0.", "date": 1484136042, "service": false, "from": {"peer_id": 265204902, "id": "$01000000a6b4ce0f77616c38a95a2f29", "peer_type": "user", "print_name": "Chat_Wars", "username": "ChatWarsBot", "flags": 1, "first_name": "Chat Wars", "last_name": ""}, "to": {"peer_id": 314226847, "id": "$010000009fb8ba120b62fa795ec02cf3", "when": "2017-01-11 20:26:06", "peer_type": "user", "phone": "79671494739", "print_name": "Mike_Udkovsky", "username": "mikeud", "flags": 524289, "first_name": "Mike", "last_name": "Udkovsky"}, "unread": false}'
2017-01-11 20:27:05,292 Received Message: "{"event": "read", "reply_id": "02000000b110f70ac6000000000000000000000000000000", "id": "02000000b110f70acd000000000000000000000000000000", "flags": 256, "out": false, "from": {"peer_id": 314998971, "id": "$01000000bb80c61256f8c2ffc2450486", "when": "2017-01-11 20:04:52", "peer_type": "user", "print_name": "ПОКРАС_ОБОЕВ", "username": "pokras_oboev", "flags": 1, "first_name": "ПОКРАС", "last_name": "ОБОЕВ"}, "to": {"peer_id": 183963825, "id": "$02000000b110f70a0000000000000000", "peer_type": "chat", "title": "Масонская ложа 🇨🇾 🇮🇲 🇻🇦 🇬🇵 🇪🇺", "print_name": "Масонская_ложа_🇨🇾_🇮🇲_🇻🇦_🇬🇵_🇪🇺", "admin": {"peer_id": 0, "id": "$01000000000000000000000000000000", "peer_type": "user", "print_name": "user#0"}, "flags": 65537, "members_num": 5}, "unread": false, "text": "ладно: ⚔ Атака, 🇬🇵", "date": 1484136045, "service": false}
{"event": "read", "id": "01000000a6b4ce0fca0000000000000077616c38a95a2f29", "flags": 256, "out": false, "text": "Ты приготовился к атаке. Ближайшее сражение через 59минут. Жди результата.", "date": 1484136042, "service": false, "from": {"peer_id": 265204902, "id": "$01000000a6b4ce0f77616c38a95a2f29", "peer_type": "user", "print_name": "Chat_Wars", "username": "ChatWarsBot", "flags": 1, "first_name": "Chat Wars", "last_name": ""}, "to": {"peer_id": 314226847, "id": "$010000009fb8ba120b62fa795ec02cf3", "when": "2017-01-11 20:26:06", "peer_type": "user", "phone": "79671494739", "print_name": "Mike_Udkovsky", "username": "mikeud", "flags": 524289, "first_name": "Mike", "last_name": "Udkovsky"}, "unread": false}"
2017-01-11 20:27:05,293 Received message could not be parsed.
Message:>{"event": "read", "reply_id": "02000000b110f70ac6000000000000000000000000000000", "id": "02000000b110f70acd000000000000000000000000000000", "flags": 256, "out": false, "from": {"peer_id": 314998971, "id": "$01000000bb80c61256f8c2ffc2450486", "when": "2017-01-11 20:04:52", "peer_type": "user", "print_name": "ПОКРАС_ОБОЕВ", "username": "pokras_oboev", "flags": 1, "first_name": "ПОКРАС", "last_name": "ОБОЕВ"}, "to": {"peer_id": 183963825, "id": "$02000000b110f70a0000000000000000", "peer_type": "chat", "title": "Масонская ложа 🇨🇾 🇮🇲 🇻🇦 🇬🇵 🇪🇺", "print_name": "Масонская_ложа_🇨🇾_🇮🇲_🇻🇦_🇬🇵_🇪🇺", "admin": {"peer_id": 0, "id": "$01000000000000000000000000000000", "peer_type": "user", "print_name": "user#0"}, "flags": 65537, "members_num": 5}, "unread": false, "text": "ладно: ⚔ Атака, 🇬🇵", "date": 1484136045, "service": false}
{"event": "read", "id": "01000000a6b4ce0fca0000000000000077616c38a95a2f29", "flags": 256, "out": false, "text": "Ты приготовился к атаке. Ближайшее сражение через 59минут. Жди результата.", "date": 1484136042, "service": false, "from": {"peer_id": 265204902, "id": "$01000000a6b4ce0f77616c38a95a2f29", "peer_type": "user", "print_name": "Chat_Wars", "username": "ChatWarsBot", "flags": 1, "first_name": "Chat Wars", "last_name": ""}, "to": {"peer_id": 314226847, "id": "$010000009fb8ba120b62fa795ec02cf3", "when": "2017-01-11 20:26:06", "peer_type": "user", "phone": "79671494739", "print_name": "Mike_Udkovsky", "username": "mikeud", "flags": 524289, "first_name": "Mike", "last_name": "Udkovsky"}, "unread": false}<
Traceback (most recent call last):
  File "C:\Users\Nonamed\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pytg\receiver.py", line 182, in _add_message
    json_dict = json.loads(text)
  File "C:\Users\Nonamed\AppData\Local\Programs\Python\Python36-32\lib\json\__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "C:\Users\Nonamed\AppData\Local\Programs\Python\Python36-32\lib\json\decoder.py", line 342, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 826)

Your environment

latest telegram-cli from master branch under Windows 10 (cygwin64 or unix subsystem), Python 3.6, pytg 0.4.10

Steps to reproduce

1. Call status_online method after every (yeld)
2. Wait for a long time (3 hours+).

Turn on debugging output (see README), what does it tell you?

Nothing except that I have shown above (it's debug log fragments).

My code looks like this:

def message_loop(options, sender):
    while not stopped:
        try:
            msg = (yield) # <----- EXCEPTION !!!
            sender.status_online(result_timeout=10) 
            ...
        except GeneratorExit:
            pass
        except KeyboardInterrupt:
            pass
        else:
            pass

Ah, sorry. I can see the reason: The first received message are actually two seperate. ...false}\n{"event":... Because the json root node is already closed } thats all the data it expects. For some reason the string contains another message as well, after the line break \n

Maybe the messages with \n could just splitted into two (probably in _add_message in receiver.py) But I'm not sure if that could split linebreaks in messages too?

I have the same problem. Events like "read" / "unread" come in such a problematic format

If someone want to contribute a fix, pull request are welcome!