UnicodeDecodeError in python-mpd2 (on Windows 10)

[handsomegui]

Hi, i found a strange problem, here below is the info:

OS: Windows 10 Pro MPD: 0.20.16 https://www.musicpd.org/download/win32/0.20.16/mpd.exe python: 2.7.14 64bit python-mpd2: 0.5.5

handler = mpd.MPDClient(use_unicode=True)
handler.timeout = 10
handler.connect("127.0.0.1", 6600)

# ...

artistList = handler.list('Artist')

Traceback (most recent call last):
  File "C:/Users/zhenwei/.code/replayer/replayer.py", line 5707, in initUI
    artistList = handler.list('Artist')
  File "C:\Python27\lib\site-packages\mpd.py", line 629, in decorator
    return wrapper(self, name, args, bound_decorator(self, returnValue))
  File "C:\Python27\lib\site-packages\mpd.py", line 254, in _execute
    return retval()
  File "C:\Python27\lib\site-packages\mpd.py", line 623, in decorator
    return function(self, *args, **kwargs)
  File "C:\Python27\lib\site-packages\mpd.py", line 398, in _fetch_list
    return self._wrap_iterator(self._read_list())
  File "C:\Python27\lib\site-packages\mpd.py", line 374, in _wrap_iterator
    return list(iterator)
  File "C:\Python27\lib\site-packages\mpd.py", line 318, in _read_list
    for key, value in self._read_pairs():
  File "C:\Python27\lib\site-packages\mpd.py", line 314, in _read_pairs
    pair = self._read_pair(separator)
  File "C:\Python27\lib\site-packages\mpd.py", line 302, in _read_pair
    line = self._read_line()
  File "C:\Python27\lib\site-packages\mpd.py", line 284, in _read_line
    line = decode_str(line)
  File "C:\Python27\lib\site-packages\mpd.py", line 33, in decode_str
    return s.decode("utf-8")
  File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 13: invalid start byte

Any help is warmly appreciated.

[rnixx]

Do you have the same problem when using master instead of 0.5.5?

[Mic92]

You can also test 1.0.0 instead, that was added some days ago. If you could dump traffic with wireshark it would be also helpful.

[Mic92]

Is the bug similar to what you have been discovered before for files? https://github.com/Mic92/python-mpd2/issues/49

[handsomegui]

Do you have the same problem when using master instead of 0.5.5?

Yes. After i pip install --upgrade python-mpd2==1.0.0

The problem remain :(

[handsomegui]

You can also test 1.0.0 instead, that was added some days ago. If you could dump traffic with wireshark it would be also helpful.

I will try to learn wireshark tool usage. Thank you.

[handsomegui]

Is the bug similar to what you have been discovered before for files? #49

I guess it might be the same issue. If this is true, then the problem was from the MPD upstream.

[handsomegui]

New found:

After i move away a folder(which contains 825 files) from music_directory to a place outside, the above problem disappeared.

So it should be caused by some special characters/encoding in the artist meta info.

[handsomegui]

I also report this issue to MPD repo.

https://github.com/MusicPlayerDaemon/MPD/issues/231

[Mic92]

@handsomegui you should provide MPD more specific information, what files causes an error. Otherwise they will not be able to fix this.

[handsomegui]

@Mic92 Thanks for the hint.

I have a lot of files under the music_directory. I have no clue to find out the file which contained the error encoding artist info.

I try to print out the unable decoded string python-mpd2 reported by add one print line in the base.py file as below image.

It shows the error decoded artist info as below:_

DEBUG INFO: OK

DEBUG INFO: Artist: 

DEBUG INFO: Artist: ????

DEBUG INFO: Artist: ????????????_??????????¼??

DEBUG INFO: Artist: ????????л??˳??

DEBUG INFO: Artist: ??????ȫ???ݳ?

DEBUG INFO: Artist: ?????��????ƽ?

'utf8' codec can't decode byte 0xc0 in position 13: invalid start byte

Could someone help me to find out the file which caused the error? Thanks.

[Mic92]

You can track this down using:

def decode_str(s):
    try:
        return s.decode("utf-8")
    except Exception as e:
        import pdb; pdb.set_trace()

then you can print s to see what is going wrong.

[handsomegui]

@Mic92 Thank you very much.

Here below is what i got:

--Return--
> c:\python27\lib\site-packages\mpd\base.py(52)decode_str()->None
-> import pdb; pdb.set_trace()
(Pdb) s
> c:\python27\lib\site-packages\mpd\base.py(528)_read_line()
-> if not line.endswith("\n"):
(Pdb) s
AttributeError: "'NoneType' object has no attribute 'endswith'"
> c:\python27\lib\site-packages\mpd\base.py(528)_read_line()
-> if not line.endswith("\n"):
(Pdb) s
--Return--
> c:\python27\lib\site-packages\mpd\base.py(528)_read_line()->None
-> if not line.endswith("\n"):
(Pdb) s
AttributeError: "'NoneType' object has no attribute 'endswith'"
> c:\python27\lib\site-packages\mpd\base.py(548)_read_lines()
-> line = self._read_line()
(Pdb) s
--Return--
> c:\python27\lib\site-packages\mpd\base.py(548)_read_lines()->None
-> line = self._read_line()
(Pdb) s
AttributeError: "'NoneType' object has no attribute 'endswith'"
> c:\python27\lib\site-packages\mpd\base.py(226)_parse_pairs()
-> for line in lines:
(Pdb) s
--Return--
> c:\python27\lib\site-packages\mpd\base.py(226)_parse_pairs()->None
-> for line in lines:
(Pdb) s
AttributeError: "'NoneType' object has no attribute 'endswith'"
> c:\python27\lib\site-packages\mpd\base.py(286)_parse_list()
-> for key, value in self._parse_pairs(lines):
(Pdb) s
--Return--
> c:\python27\lib\site-packages\mpd\base.py(286)_parse_list()->None
-> for key, value in self._parse_pairs(lines):
(Pdb) s
AttributeError: "'NoneType' object has no attribute 'endswith'"
> c:\python27\lib\site-packages\mpd\base.py(567)_wrap_iterator()
-> return list(iterator)
(Pdb) s
--Return--
> c:\python27\lib\site-packages\mpd\base.py(567)_wrap_iterator()->None
-> return list(iterator)
(Pdb) s
AttributeError: "'NoneType' object has no attribute 'endswith'"
> c:\python27\lib\site-packages\mpd\base.py(378)command_callback()
-> res = self._wrap_iterator(res)
(Pdb) s
--Return--
> c:\python27\lib\site-packages\mpd\base.py(378)command_callback()->None
-> res = self._wrap_iterator(res)
(Pdb) s
AttributeError: "'NoneType' object has no attribute 'endswith'"
> c:\python27\lib\site-packages\mpd\base.py(480)_execute()
-> return retval()
(Pdb) s
--Return--
> c:\python27\lib\site-packages\mpd\base.py(480)_execute()->None
-> return retval()
(Pdb) s
AttributeError: "'NoneType' object has no attribute 'endswith'"
> c:\python27\lib\site-packages\mpd\base.py(388)mpd_command()
-> return wrapper(self, name, args, callback)
(Pdb) s
--Return--
> c:\python27\lib\site-packages\mpd\base.py(388)mpd_command()->None
-> return wrapper(self, name, args, callback)
(Pdb) s
AttributeError: "'NoneType' object has no attribute 'endswith'"
> c:\users\zhenwei\.code\replayer\replayer.py(5744)initUI()
-> artistList = handler.list('Artist')
(Pdb) s
--Return--
> c:\users\zhenwei\.code\replayer\replayer.py(5744)initUI()->None
-> artistList = handler.list('Artist')
(Pdb) s
AttributeError: "'NoneType' object has no attribute 'endswith'"
> c:\users\zhenwei\.code\replayer\replayer.py(5727)__init__()
-> self.initUI(self.font)
(Pdb) s
--Return--
> c:\users\zhenwei\.code\replayer\replayer.py(5727)__init__()->None
-> self.initUI(self.font)
(Pdb) s
AttributeError: "'NoneType' object has no attribute 'endswith'"
> c:\users\zhenwei\.code\replayer\replayer.py(1376)init_ui()
-> self.libraryView = Library(self.fontValue)
(Pdb) s
--Return--
> c:\users\zhenwei\.code\replayer\replayer.py(1376)init_ui()->None
-> self.libraryView = Library(self.fontValue)
(Pdb) s
AttributeError: "'NoneType' object has no attribute 'endswith'"
> c:\users\zhenwei\.code\replayer\replayer.py(548)__init__()
-> self.init_ui()
(Pdb) s
--Return--
> c:\users\zhenwei\.code\replayer\replayer.py(548)__init__()->None
-> self.init_ui()
(Pdb) s
AttributeError: "'NoneType' object has no attribute 'endswith'"
> c:\users\zhenwei\.code\replayer\replayer.py(6749)<module>()
-> mw = MW()
(Pdb) s
--Return--
> c:\users\zhenwei\.code\replayer\replayer.py(6749)<module>()->None
-> mw = MW()
(Pdb) s
Traceback (most recent call last):
  File "C:/Users/zhenwei/.code/replayer/replayer.py", line 6749, in <module>
    mw = MW()
  File "C:/Users/zhenwei/.code/replayer/replayer.py", line 548, in __init__
    self.init_ui()
  File "C:/Users/zhenwei/.code/replayer/replayer.py", line 1376, in init_ui
    self.libraryView = Library(self.fontValue)
  File "C:/Users/zhenwei/.code/replayer/replayer.py", line 5727, in __init__
    self.initUI(self.font)
  File "C:/Users/zhenwei/.code/replayer/replayer.py", line 5744, in initUI
    artistList = handler.list('Artist')
  File "C:\Python27\lib\site-packages\mpd\base.py", line 388, in mpd_command
    return wrapper(self, name, args, callback)
  File "C:\Python27\lib\site-packages\mpd\base.py", line 480, in _execute
    return retval()
  File "C:\Python27\lib\site-packages\mpd\base.py", line 378, in command_callback
    res = self._wrap_iterator(res)
  File "C:\Python27\lib\site-packages\mpd\base.py", line 567, in _wrap_iterator
    return list(iterator)
  File "C:\Python27\lib\site-packages\mpd\base.py", line 286, in _parse_list
    for key, value in self._parse_pairs(lines):
  File "C:\Python27\lib\site-packages\mpd\base.py", line 226, in _parse_pairs
    for line in lines:
  File "C:\Python27\lib\site-packages\mpd\base.py", line 548, in _read_lines
    line = self._read_line()
  File "C:\Python27\lib\site-packages\mpd\base.py", line 528, in _read_line
    if not line.endswith("\n"):
AttributeError: 'NoneType' object has no attribute 'endswith'
Exception AttributeError: "'NoneType' object has no attribute 'path'" in <function _remove at 0x0000000004A9AF98> ignored

Process finished with exit code 1

Any useful info for debug? Could you point me more? Thanks.

[Mic92]

s is also a key in pdb, you did not print anything but just skipped execution. Use print(s) instead.

[handsomegui]

@Mic92

Sorry for my poor knowledge on this.

Here is what i get:

Artist: ?????��????ƽ?

--Return--
> c:\python27\lib\site-packages\mpd\base.py(53)decode_str()->None
-> import pdb; pdb.set_trace()
(Pdb) 

[Mic92]

Can you print the following?

print([ c for c in s])

And post the result also upstream.

[Mic92]

Maybe also share the file with upstream.

[handsomegui]

Thanks @Mic92

Here is the output:

Artist: ?????��????ƽ?

['A', 'r', 't', 'i', 's', 't', ':', ' ', '?', '?', '?', '?', '?', '\xc0', '\xa1', '?', '?', '?', '?', '\xc6', '\xbd', '?', '\n']
--Return--
> c:\python27\lib\site-packages\mpd\base.py(54)decode_str()->None
-> import pdb; pdb.set_trace()
(Pdb) 

Sorry. I still have no clue to find out the specific file :(

[handsomegui]

Unfortunately, the upstream admin closed my issue report.

[Mic92]

well, just install wireshark.

[guizhenwei]

FYI.

I finally picked out the file which caused the UTF-8 decoding error:

https://www.dropbox.com/s/f72ti62v7op6biu/KPSU.mp3?dl=0

I've reported this to MPD issues list also.