yapf failing on valid Python

[fathom-max]

I have the following line in code that makes yapf fail:

FIELD_PATTERN: str = fr'^({SEARCH_FIELD}|{SEARCH_FIELD}[ \t]+[a-z0-9])$'

It causes the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/yapf/yapflib/pytree_utils.py", line 115, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/local/lib/python3.6/lib2to3/pgen2/driver.py", line 107, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/local/lib/python3.6/lib2to3/pgen2/driver.py", line 72, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/local/lib/python3.6/lib2to3/pgen2/parse.py", line 159, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=3, value="'^({SEARCH_FIELD}|{SEARCH_FIELD}[ \\t]+[a-z0-9])$'", context=('', (76, 23))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/yapf", line 10, in <module>
    sys.exit(run_main())
  File "/usr/local/lib/python3.6/site-packages/yapf/__init__.py", line 344, in run_main
    sys.exit(main(sys.argv))
  File "/usr/local/lib/python3.6/site-packages/yapf/__init__.py", line 226, in main
    verbose=args.verbose)
  File "/usr/local/lib/python3.6/site-packages/yapf/__init__.py", line 278, in FormatFiles
    in_place, print_diff, verify, quiet, verbose)
  File "/usr/local/lib/python3.6/site-packages/yapf/__init__.py", line 305, in _FormatFile
    logger=logging.warning)
  File "/usr/local/lib/python3.6/site-packages/yapf/yapflib/yapf_api.py", line 91, in FormatFile
    verify=verify)
  File "/usr/local/lib/python3.6/site-packages/yapf/yapflib/yapf_api.py", line 129, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "/usr/local/lib/python3.6/site-packages/yapf/yapflib/pytree_utils.py", line 121, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/local/lib/python3.6/lib2to3/pgen2/driver.py", line 107, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/local/lib/python3.6/lib2to3/pgen2/driver.py", line 72, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/local/lib/python3.6/lib2to3/pgen2/parse.py", line 159, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=3, value="'^({SEARCH_FIELD}|{SEARCH_FIELD}[ \\t]+[a-z0-9])$'", context=('', (76, 23))

• I was initially running yapf 0.29, then I updated to 0.30, but the problem persisted.
• This was reproduced on another machine by a colleague.
• I use Python 3.6.8

[jaymody]

Odd, worked for me using python 3.7.5 with yapf 0.29.0 and 0.30.0. Tested this with the following code and command:

script.py

SEARCH_FIELD="hello"
FIELD_PATTERN: str = fr'^({SEARCH_FIELD}|{SEARCH_FIELD}[ \t]+[a-z0-9])$'

command

yapf script.py

[fathom-max]

@jaymody I suspect this may have to do with Python 3.6

[laoshaw]

  def divide(x, y):
       try:
            result = x / y
        except ZeroDivisionError:
            raise ValueError("divided by zero")
        else:
            print("result is", result)
        finally:
              print("done cleanup")

if __name__ == "__main__":
        divide(10, 2)
    divide(10, 0)

run yapf my.py(above) I got:

Traceback (most recent call last):
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/yapflib/pytree_utils.py", line 115, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/lib/python3.8/lib2to3/pgen2/driver.py", line 103, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/lib/python3.8/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/lib/python3.8/lib2to3/pgen2/parse.py", line 162, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=5, value='  ', context=('', (1, 0))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/yapflib/pytree_utils.py", line 121, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/lib/python3.8/lib2to3/pgen2/driver.py", line 103, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/lib/python3.8/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/lib/python3.8/lib2to3/pgen2/parse.py", line 162, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=5, value='  ', context=('', (1, 0))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/r6aa/.local/bin/yapf", line 8, in <module>
    sys.exit(run_main())
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/__init__.py", line 365, in run_main
    sys.exit(main(sys.argv))
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/__init__.py", line 125, in main
    changed = FormatFiles(
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/__init__.py", line 203, in FormatFiles
    changed |= _FormatFile(filename, lines, style_config, no_local_style,
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/__init__.py", line 226, in _FormatFile
    reformatted_code, encoding, has_change = yapf_api.FormatFile(
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/yapflib/yapf_api.py", line 94, in FormatFile
    reformatted_source, changed = FormatCode(
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/yapflib/yapf_api.py", line 147, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/yapflib/pytree_utils.py", line 127, in ParseCodeToTree
    raise e
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/yapflib/pytree_utils.py", line 125, in ParseCodeToTree
    ast.parse(code)
  File "/usr/lib/python3.8/ast.py", line 47, in parse
    return compile(source, filename, mode, flags,
  File "except1.py", line 1
    def divide(x, y):
    ^
IndentationError: unexpected indent

yes the indent error is intentional, to see if yapf can fix that. autopep8 gave no warnings and does nothing, black complains but not crash(still can not fix that simple indentation)

[kamahen]

It's not a valid Python program. You can't expect yapf to fix an invalid program.

$ python3.7 c.py
  File "c.py", line 13
    divide(10, 0)
                ^
IndentationError: unindent does not match any outer indentation level

  def divide(x, y):
       try:
            result = x / y
        except ZeroDivisionError:
            raise ValueError("divided by zero")
        else:
            print("result is", result)
        finally:
              print("done cleanup")

if __name__ == "__main__":
        divide(10, 2)
    divide(10, 0)

run yapf my.py(above) I got:

Traceback (most recent call last):
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/yapflib/pytree_utils.py", line 115, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/lib/python3.8/lib2to3/pgen2/driver.py", line 103, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/lib/python3.8/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/lib/python3.8/lib2to3/pgen2/parse.py", line 162, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=5, value='  ', context=('', (1, 0))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/yapflib/pytree_utils.py", line 121, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/lib/python3.8/lib2to3/pgen2/driver.py", line 103, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/lib/python3.8/lib2to3/pgen2/driver.py", line 71, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/lib/python3.8/lib2to3/pgen2/parse.py", line 162, in addtoken
    raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=5, value='  ', context=('', (1, 0))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/r6aa/.local/bin/yapf", line 8, in <module>
    sys.exit(run_main())
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/__init__.py", line 365, in run_main
    sys.exit(main(sys.argv))
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/__init__.py", line 125, in main
    changed = FormatFiles(
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/__init__.py", line 203, in FormatFiles
    changed |= _FormatFile(filename, lines, style_config, no_local_style,
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/__init__.py", line 226, in _FormatFile
    reformatted_code, encoding, has_change = yapf_api.FormatFile(
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/yapflib/yapf_api.py", line 94, in FormatFile
    reformatted_source, changed = FormatCode(
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/yapflib/yapf_api.py", line 147, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/yapflib/pytree_utils.py", line 127, in ParseCodeToTree
    raise e
  File "/home/r6aa/.local/lib/python3.8/site-packages/yapf/yapflib/pytree_utils.py", line 125, in ParseCodeToTree
    ast.parse(code)
  File "/usr/lib/python3.8/ast.py", line 47, in parse
    return compile(source, filename, mode, flags,
  File "except1.py", line 1
    def divide(x, y):
    ^
IndentationError: unexpected indent

yes the indent error is intentional, to see if yapf can fix that. autopep8 gave no warnings and does nothing, black complains but not crash(still can not fix that simple indentation)

[duduzai2019]

hi,how to solve it?

[bwendling]

Closed with 7c408b9d7750292760ed255f744211d1ef535668.