grammars-v4 icon indicating copy to clipboard operation
grammars-v4 copied to clipboard

UnicodeDecodeError with Python3 runtime

Open RobEin opened this issue 2 years ago • 1 comments

I have a repository with a Python 3 parser. And there are a couple of problematic files that Java target parses without error, but the Python target throws an error.

problematic files: fractions.py ftplib.py functools.py getopt.py heapq.py random.py runpy.py shlex.py statistics.py

to reproduce the error:

antlr4 -Dlanguage=Python3 PythonLexer.g4
antlr4 -Dlanguage=Python3 PythonParser.g4
pygrun --tokens Python file_input fractions.py

error message:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 15765: ordinal not in range(128)

RobEin avatar Sep 17 '23 14:09 RobEin

I found the easiest way to reproduce the error. For example the following 2-character Python code (which is a comment) causes the same error message:

More precisely: Unicode Character “Å” (U+00C5)

RobEin avatar Sep 19 '23 22:09 RobEin