"unindent does not match any outer indentation level" for valid Python syntax

[goretkin]

Describe the bug

Code that is valid python syntax gives:

error: cannot format bug_with_black_autoformatter.py: unindent does not match any outer indentation level (<tokenize>, line 4)

Oh no! 💥 💔 💥
1 file failed to reformat.

To Reproduce https://black.vercel.app/?version=main%3E&state=_Td6WFoAAATm1rRGAgAhARYAAAB0L-Wj4ADuAJRdAD2IimZxl1N_Wlws4TMyioAEHvymg6UIHhjuL4u9Oa7M3Bizxl0vPANAhEBiT8-fRu9Q-bJ6pLVlHCH7DrJ7_UM9vSx0SQl3h6nFpHrpQiw1sj4eJ1EnO4Zc179ROMi5VwkyyjB3T76GG6vJhnG6OGpwtvAsUsCFar9Bk1lcDEyDx7_El_LaKcZSGqI8A2N9PWVpU_gAd0hX5nTcl1AAAbAB7wEAAEHVIWuxxGf7AgAAAAAEWVo=

if __name__ == "__main__":
  foo = 3
  # fmt: off
  bar = [
    0, 0, 0, 0, 0, 0]
  # fmt: on
  baz = 4
  print(foo, bar, baz)

run it with $ black bug_with_black_autoformatter.py

The resulting error is:

$ black bug_with_black_autoformatter.py
error: cannot format bug_with_black_autoformatter.py: unindent does not match any outer indentation level (<tokenize>, line 4)

Oh no! 💥 💔 💥
1 file failed to reformat.

Note that it is valid python code:

$ python bug_with_black_autoformatter.py
3 [0, 0, 0, 0, 0, 0] 4

Expected behavior

Black auto-formats the code.

Environment

Reproduced in https://black.vercel.app/

[ichard26]

Black is producing invalid code. The error is misleading, while it is technically correct, it's caused by Black itself. Usually Black would emit a helpful INTERNAL ERROR: Black produced invalid code error, but that's been broken for a while now. I'll open an issue soon if it doesn't exist (and fix it within the month).

if __name__ == "__main__":
    foo = 3
    # fmt: off
  bar = [
    0, 0, 0, 0, 0, 0]
    # fmt: on
    baz = 4
    print(foo, bar, baz)

(Just so it's clear, Black is formatting the code twice internally in this case. The second format pass is being fed invalid code produced by the first pass.)

[JelleZijlstra]

So the reason this happen is that Black (correctly) reindents the code to use 4-space indents, except within the # fmt: off part, because the user asked us not to format it. So the code it tries to output then has a syntax error from the remaining 2-space indents.

I don't see a way we can format this successfully. We're not going to keep the 2-space indents, and we're also not going to re-indent the # fmt: off block. So I think the right behavior would be for Black to fail, ideally with a clearer error message.

[luisrayas3]

Found this as well, I agree this should still be an error but the message is not great. The user should be instructed to fix the indentation manually

[JelleZijlstra]

#3702 and #3705 report two similar cases with slightly different symptoms.

I still don't think there is a way we can format these cases correctly, but we should give a better error.

[Feuermurmel]

Just my two cents, maybe my input helps:

I stumbled on this with a function that had all it's code indented by 8 spaces, including a # fmt: off section, after moving the code into a new function from another function where it was indented by that amount. I was confused for some time.

I see there are good reasons agains touching the # fmt: off section at all. OTOH, my expectation in my case was that black would simply outdent the whole body of the function by 4 spaces, including the # fmt comments and everything in-between. Manually doing that was of course trivial, once I knew what the issue was.

But I think printing a better error message is almost as good AFAICT much simpler to implement as less contentious.