black fails to parse a seemingly fine Python file in safe mode, succeeds in fast mode and in version 18.9

[raphj]

trafficstars

Describe the bug

It looks like a regression. We get the following error by running black --safe with Black versions 20.8b1 and 19.10b0 on file https://raw.githubusercontent.com/tracim/tracim/2e84f5d6427f06f9b64652dc042f168b7c76f8b1/backend/tracim_backend/lib/core/content.py

error: cannot format content.py: cannot use --safe with this file; failed to parse source file.  AST error message: invalid syntax (<unknown>, line 83)
Oh no! 💥 💔 💥
1 file failed to reformat.

black --fast successfully reformats the file. Then, black --safe works on the resulting file.

We have not found out why. Black 18.9b0 as provided in Debian Buster does not show this issue.

To Reproduce Steps to reproduce the behavior:

$ cd /tmp
$ wget https://raw.githubusercontent.com/tracim/tracim/2e84f5d6427f06f9b64652dc042f168b7c76f8b1/backend/tracim_backend/lib/core/content.py
$ python3 -m venv /tmp/venv
$ source /tmp/venv/bin/activate
$ python3 --version
Python 3.7.3
$ pip install black
$ rehash
$ which black
/tmp/venv/bin/black
$ black --version
black, version 20.8b1
$ black content.py
error: cannot format content.py: cannot use --safe with this file; failed to parse source file.  AST error message: invalid syntax (<unknown>, line 83)
Oh no! 💥 💔 💥
1 file failed to reformat.
$ black --fast content.py
reformatted content.py
All done! ✨ 🍰 ✨
1 file reformatted.
$ black content.py
All done! ✨ 🍰 ✨
1 file left unchanged.

Expected behavior: black should successfully parse this file with the --safe flag Environment (please complete the following information):

• Version: 20.8b1 and 19.10b0 (not 18.9b0)
• OS and Python version: Debian GNU/Linux Buster, Python 3.7.3

Does this bug also happen on master? Yes

Additional context: None yet

[asottile]

the underlying error:

(Pdb) p e
SyntaxError('misplaced type annotation', ('<unknown>', 799, 55, '        yield depot_stored_file._file_path  # type: str\n'))

https://github.com/tracim/tracim/blob/2e84f5d6427f06f9b64652dc042f168b7c76f8b1/backend/tracim_backend/lib/core/content.py#L799

that's an invalid type annotation so it's tripping up typed-ast3 and falling back to typed-ast27 which is choking on your python3-syntax type annotations

in python3.8+ this uses standard library typed ast which appears to ignore this comment

[raphj]

Thank you for the investigation. This annotation is still present in the code reformatted by black --fast. Do you know why black --safe works on the reformatted file? is typed-ast only called if reformatting is necessary?

[raphj]

Duplicate of https://github.com/psf/black/issues/1461 then?

[asottile]

no idea, I don't maintain or even really use black but I found your issue title interesting

[raphj]

Ok :-)

To Black mantainers: would printing errors from typed-ast make sense?

[JelleZijlstra]

Sounds like this is because we use typed-ast to implement the safety check, and typed-ast doesn't like a type comment in that position.

[Jordan-Cottle]

I've encountered this myself recently while formatting a large file that has a ton of large blocks of comments that frequently contain example data. Black would trip up on any lines that included something like # type: foo.

A minimal example that fails for me is a file with exactly the content

# type: foo

The blank line at the start is important for some reason. I think it's because files that don't have any formatting changes to make won't go through the AST validation, but I've not reviewed the black code enough to confidently state that.

$ cat t.py

# type: foo
$ black t.py 
error: cannot format t.py: cannot use --safe with this file; failed to parse source file AST: invalid syntax (<unknown>, line 2)
This could be caused by running Black with an older Python version that does not support new syntax used in your source file.

Oh no! 💥 💔 💥
1 file failed to reformat.

[samhinshaw]

I have come across the same error running black on a file. I had a comment block illustrating an interface using TypeScript syntax:

# interface Item {
#     id: foo;
#     type: bar;
# }

This produced the following error message:

error: cannot format <filepath>: cannot use --safe with this file; failed to parse source file AST: invalid syntax (<unknown>, line 125)
This could be caused by running Black with an older Python version that does not support new syntax used in your source file.

Oh no! 💥 💔 💥
1 file failed to reformat.

[ichard26]

@samhinshaw please try black 23.3.0. It should handle invalid type comments without blowing up.

[samhinshaw]

It works! Thanks so much 🤙

[JelleZijlstra]

As @ichard26 said, this should be fixed now: we parse without type comments if parsing with type comments fails.