Exception occurs under Python 3.8

[stevenleeS0ht]

yapf seems not support python3.8

versions:

python -- 3.8.0 yapf -- 0.29.0

When I reformat the source tree which definitely no error, the following error message occurs:

$ yapf -pri .
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/yapf/yapflib/pytree_utils.py", line 115, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/lib/python3.8/lib2to3/pgen2/driver.py", line 103, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/lib/python3.8/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/lib/python3.8/lib2to3/pgen2/parse.py", line 162, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=22, value='=', context=('', (25, 16))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/lib/python3.8/site-packages/yapf/__init__.py", line 298, in _FormatFile
    reformatted_code, encoding, has_change = yapf_api.FormatFile(
  File "/usr/lib/python3.8/site-packages/yapf/yapflib/yapf_api.py", line 85, in FormatFile
    reformatted_source, changed = FormatCode(
  File "/usr/lib/python3.8/site-packages/yapf/yapflib/yapf_api.py", line 129, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "/usr/lib/python3.8/site-packages/yapf/yapflib/pytree_utils.py", line 121, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/lib/python3.8/lib2to3/pgen2/driver.py", line 103, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/lib/python3.8/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/lib/python3.8/lib2to3/pgen2/parse.py", line 162, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=22, value='=', context=('', (25, 16))
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/yapf", line 11, in <module>
    load_entry_point('yapf==0.29.0', 'console_scripts', 'yapf')()
  File "/usr/lib/python3.8/site-packages/yapf/__init__.py", line 344, in run_main
    sys.exit(main(sys.argv))
  File "/usr/lib/python3.8/site-packages/yapf/__init__.py", line 216, in main
    changed = FormatFiles(
  File "/usr/lib/python3.8/site-packages/yapf/__init__.py", line 274, in FormatFiles
    changed |= future.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
lib2to3.pgen2.parse.ParseError: xxx/xxx.py

[stevenleeS0ht]

It probably caused by the new operator ":=" introduced in python 3.8

[stevenleeS0ht]

@gwelymernans , Please have a look.

[bwendling]

@stevenleeS0ht This is most likely due to lib2to3 not being able to handle := yet. We rely upon it for our parsing, and there's not much we can do until it's been updated.

[tshirtman]

Bummer, i was keen on using it on my codebase. Doesn't work with python 3.8.2. Seems it's in progress on python's side https://bugs.python.org/issue36541 probably need to check with the new 3.9 beta.

[dmaple]

It looks like lib2to3 was fixed in Dec 14, 2020. https://bugs.python.org/issue36541

[wookayin]

Any updates on this?