Unexpected incompatibility with reorder-python-imports

[maxwell-k]

Describe the bug

It is impossible to use Black 24.1.0 alongside reorder-python-imports. Using the example from #1872 , reorder-python-imports produces:

"""Module docstring."""
import typing 

While formatting with black and default arguments produces the following:

"""Module docstring."""

import typing

This new behaviour makes the two tools incompatible. The author of reorder-python-imports suggests this is an issue with Black.

Is it possible to change Black to work better with reorder-python-imports please? I have used both tools together for a long time.

Environment

black, 24.1.0 (compiled: yes)
Python (CPython) 3.12.1

Also checked on the online formatter.

Additional context

A little bit of context from the Black changelog is that Black 24 and above will:

Enforce newline after module docstrings (#3932, #4028)

1872 was the original Black issue, and changes were implemented in 3932; 4028 doesn't relate to imports.

[JelleZijlstra]

It's unfortunate that nobody reported this while the feature was in the preview style and in the alphas. Our guarantee is that we'll keep the style the same for the rest of the year, so we can't change it now.

I think it should be Black's job to decide on whitespace outside the import block, not reorder-python-imports's. Unfortunately that means I don't have a good resolution for you.

[cooperlees]

I'll second Jelle here and state that an import re-ordering tool should respect the syntax it was given when it started it's run.

Here, if there is white space between an import and a docstring, it should be maintained. It's not a formatting tool, the white space has nothing to do with fixing your imports, so I feel this tool is going outside it's scope.

[maxwell-k]

I agree this is unfortunate. I really like the other changes in the current style that was adopted with 24.1.0. I didn't know about the guarantee - I do now . I hadn't thought to test the preview style or alphas; probably because I didn't realise this is an uncommon combination of tools. Again I've learnt something, thank you.

I'm sorry if this issue is not the right place for this conversation; the original report is closed, locked as resolved and I'm unable to comment.

For the code I'm working with today # fmt: skip after the closing """ on the problem docstrings is an acceptable workaround.

Reading using black with other tools again; I suggest a simple solution might be to document that these two tools are not compatible, that what I'm trying to do is unsupported.

I think it should be Black's job to decide on whitespace outside the import block, not reorder-python-imports's. Unfortunately that means I don't have a good resolution for you.

Here, if there is white space between an import and a docstring, it should be maintained. It's not a formatting tool, the white space has nothing to do with fixing your imports, so I feel this tool is going outside it's scope.

To get to an important point, now I have seen:

1. the maintainer of the import reordering tool explain that the issue is with the code formatting tool and
2. the maintainers of the code formatting tool explain that the problem is with the import reordering tool

I don't want to comment either way. Today, the two tools are incompatible. If future versions are to be compatible one or both must change. The import reordering tool appears to have ruled out changes. Are changes a possibility for this code formatting tool?

Thank you for you help! Black is a great piece of software.

[JelleZijlstra]

I realize you're in an unfortunate situation where two opinionated tools disagree in an incompatible way, and I wish we'd have known about it during the preview period. I won't rule out changing our style here in the future if there's a good reason to do so, but I think that's unlikely. It's definitely within Black's purview to control whitespace after the module docstring.

Possible workarounds:

• In your CI, ensure that your code is clean if you use reorder-python-imports then Black (or the other way around), instead of running only one of the tools.
• Use a different tool for import sorting, e.g. isort, usort, or Ruff's equivalent of isort.

[maxwell-k]

The only reason to change that I am aware of is to make Black compatible with reorder-python-imports again. Is that a good reason or not? I'll let you decide.

I agree both are opinionated tools with a specific style. I should perhaps make it explicit here that the style implemented by reorder-python-imports has "a single aim: reduce merge conflicts" with a documented rationale. A short example is:

-from typing import Dict, List
+from typing import Dict
+from typing import List

The other functionality in reorder-python-imports is also available in other import sorting tools. I looked at isort and ruff; usort is new to me, thank you for mentioning it. I am not aware of a tool other than reorder-python-imports that implements this "reduce merge conflicts" import style.

isort is the only documented compatible import ordering tool for black. Does that imply anything about future compatibilty?

The ignore comments ( # fmt: skip after the closing """ ) I mentioned above are OK for a short term workaround. Longer term, if neither tool is going to change, you're right I'll need to carefully reconsider my tool choices.

Edited to add: thanks again for dealing with this unfortunate incompatibility carefully. I appreciate your help.

[JelleZijlstra]

Might be worth asking Ruff to implement the one-import-per-line style as an option. (I searched their issue tracker but didn't find an existing issue asking for it.)

[zachborboacryptofi]

maybe just use isort with something like:

[tool.isort]
profile = "black"
force_single_line = "true"

[adamchainz]

🤷 This is a real shame, I really like the speed and low-configurability of reorder-python-imports. I do also think its rearrangement of non-import newlines is outside its wheelhouse.

FWIW, below is the small patch that makes reorder-python-imports compatible. Debating whether it's worth forking to make an “always Black-compatible” version...

diff --git reorder_python_imports.py reorder_python_imports.py
index d66dc4b..3de766f 100644
--- reorder_python_imports.py
+++ reorder_python_imports.py
@@ -94,10 +94,10 @@ def partition_source(src: str) -> tuple[str, list[str], str, str]:
             pre_import = False
             chunks.append((CodeType.IMPORT, s))
         elif token_type is Tok.NEWLINE:
-            if s.isspace():
-                tp = CodeType.NON_CODE
-            elif pre_import:
+            if pre_import:
                 tp = CodeType.PRE_IMPORT_CODE
+            elif s.isspace():
+                tp = CodeType.NON_CODE
             else:
                 tp = CodeType.CODE
 
diff --git tests/reorder_python_imports_test.py tests/reorder_python_imports_test.py
index c61c39a..4db72fd 100644
--- tests/reorder_python_imports_test.py
+++ tests/reorder_python_imports_test.py
@@ -47,7 +47,7 @@ def test_tokenize_can_match_strings(s):
 @pytest.mark.parametrize(
     's',
     (
-        pytest.param('', id='trivial'),
+        pytest.param('\n', id='trivial'),
         pytest.param('#!/usr/bin/env python\n', id='shebang'),
         pytest.param('# -*- coding: UTF-8 -*-\n', id='source encoding'),
         pytest.param('  # coding: UTF-8\n', id='source encoding indented'),
@@ -190,17 +190,21 @@ def test_partition_source_imports_only(s, expected):
     assert nl == '\n'
 
 
-def test_partition_source_before_removes_newlines():
+def test_partition_source_before_leaves_newlines():
     before, imports, after, nl = partition_source(
         '# comment here\n'
         '\n'
-        '# another comment here\n',
+        '# another comment here\n'
+        '\n'
+        'import os\n'
     )
     assert before == (
         '# comment here\n'
+        '\n'
         '# another comment here\n'
+        '\n'
     )
-    assert imports == []
+    assert imports == ['import os\n']
     assert after == ''
     assert nl == '\n'
 

[tolomea]

The guy who maintains reorder-python-imports is very "my way or the highway", I suggest you take that advice and use isort or the fork suggestion @adamchainz made. The place I work already dropped reorder-python-imports not because it didn't work for us but specifically because of his attitude. If one of you does fork it please let me know, we would definitely consider switching to an import sorter that had the same sort of considered opination that Black has.

[adamchainz]

I’ve decided to migrate my projects to isort with the configuration:

[tool.isort]
add_imports = [
    "from __future__ import annotations"
]
force_single_line = true
profile = "black"

isort is still sufficiently fast for me and it does have a goal of supporting Black.

[maxwell-k]

I noticed that an "unexpected comment" will prevent the incompatible behaviour; neither tool reformats code like the example below:

#!/usr/bin/env python3
# unexpected comment
"""Module docstring."""
import typing

[Achimh3011]

Hmm, black makes a change so it is incompatible with python-reorder-importsand tells us "tough luck" for those who use both.

I'll move on to ruff.

[nicoddemus]

I suggest for this issue to be closed, with the solution being using isort: https://github.com/psf/black/issues/4175#issuecomment-1936708759

Perhaps it can be pinned for some time so others can find it quickly. 👍

[pbhuss]

Hmm, black makes a change so it is incompatible with python-reorder-importsand tells us "tough luck" for those who use both.

I'll move on to ruff.

@Achimh3011 ruff won't help you here either -- they've adopted the same incompatible rule into their 2024.2 style https://github.com/astral-sh/ruff/issues/7995

[Achimh3011]

Unfortunately, you're right. I wanted to leave the quarrels behind by moving to a single tool, but the missing single imports feature of the import sorter of ruff spoils it for me. Either I stick to an older version of black or deactivate automatic code formatting.

I have no opinion on the newline itself, it's only the incompatibility that causes the pain.

[zstyblik]

Perhaps this is a good place to ask, because I have exactly the same problem Black Vs. python-reorder-imports. Is it possible to somehow disable Enforce newline after module docstrings? I went through related GH issues, but I haven't found any way to do so - and perhaps there isn't one.

Thanks

[JelleZijlstra]

It is not possible.

[zstyblik]

@JelleZijlstra understood :\ Thanks.

[JelleZijlstra]

Going to close this as we are not going to make any change to Black.

Summary:

• Black 24+ and reorder-python-imports have incompatible opinions on whether there should be a blank line after a docstring. Neither tool is likely to change its behavior, and neither provides a configuration option controlling the behavior.
• Workarounds include:
  • Instead of reorder-python-imports, use isort with a specific config (https://github.com/psf/black/issues/4175#issuecomment-1915522848, https://github.com/psf/black/issues/4175#issuecomment-1936708759)
  • Patching reorder-python-imports (https://github.com/psf/black/issues/4175#issuecomment-1925312438)
  • Adding a comment (https://github.com/psf/black/issues/4175#issuecomment-1943873280)

[maxwell-k]

Thank you for dealing with this issue so patiently @JelleZijlstra !

I've learnt from all the helpful comments; and I'm happy even if in the end the two tools are incompatible in the end.

I agree completely with your summary, one very minor point is that the third link "Adding a comment" should probably be this coment: https://github.com/psf/black/issues/4175#issuecomment-1943873280

Thanks again.

[wimglenn]

If one of you does fork it please let me know, we would definitely consider switching to an import sorter that had the same sort of considered opination that Black has.

Reluctantly, I went ahead and made the fork

• https://github.com/wimglenn/reorder-python-imports-black
• https://pypi.org/project/reorder-python-imports-black/

Hopefully in the future these projects can reconcile their differences and the fork can cease to exist.