exception while during frosted

[MirkoDziadzka]

Hi

If I run frosted over a bigger codebase, I get the following exception. Unfortunately it does not tell me which file gives the problem (which it should) but simply excepts.

$ frosted --verbose -i E101 -i W101 -i E307 -i E103 -i E301 -i E302 -r . Traceback (most recent call last): File "MYPATH/testenv/bin/frosted", line 9, in load_entry_point('frosted==1.4.1', 'console_scripts', 'frosted')() File "MYPATH/testenv/lib/python2.7/site-packages/frosted/main.py", line 38, in main warnings = check_recursive(file_names, *_arguments) File "MYPATH/testenv/lib/python2.7/site-packages/frosted/api.py", line 152, in check_recursive warnings += check_path(source_path, reporter, settings_path=None, *_setting_overrides) File "MYPATH/testenv/lib/python2.7/site-packages/frosted/api.py", line 133, in check_path return check(codestr, filename, reporter, settings_path, *_setting_overrides) File "MYPATH/testenv/lib/python2.7/site-packages/frosted/api.py", line 114, in check w = checker.Checker(tree, filename, None, ignore_lines=_noqa_lines(codeString), *_active_settings) File "MYPATH/testenv/lib/python2.7/site-packages/frosted/api.py", line 40, in _noqa_lines g = tokenize.generate_tokens(StringIO(str(codeString)).readline) # tokenize the string UnicodeDecodeError: 'utf8' codec can't decode byte 0xf8 in position 6783: invalid start byte

[carlio]

I have run into the same problem. I can't speak for @MirkoDziadzka but my problem comes from a non-utf-8 encoded file.

Specifically this line of this file: https://github.com/carlio/Pillow/blob/master/PIL/WalImageFile.py#L78 which is encoded using iso-8859-1.

You can also reproduce this very easily by creating a file with the following contents:

# ü

Then save it with ISO-8859-1 encoding (if you use vim, you can :w ++enc=iso-8859-1 filename.py) and run frosted filename.py.

This does not happen in pyflakes; I wasn't able to figure out exactly why but I think it's just by accident rather than design - in pyflakes the unicode codeString is never manipulated / str'd.