cvxpy
cvxpy copied to clipboard
cvxpy
`isort` introduces bugs
Describe the bug
Following the CONTRIBUTING guide, I cloned the repository and ran isort. I expected it to not make any change, as the code is fresh from repository. However, three files were changed:
cvxpy/__init__.py
cvxpy/reductions/__init__.py
cvxpy/reductions/dcp2cone/atom_canonicalizers/__init__.py
After that, when I ran
pytest cvxpy/tests/test_problem.py
I got an error: ImportError: cannot import name 'InverseData' from partially initialized module 'cvxpy.reductions' (most likely due to a circular import).
The error was gone after I reverted the changes made by isort.
To Reproduce
-
clone the repository.
-
create a new virtual environment:
> virtualenv .venv39a --python=C:\Users\user\AppData\Local\Programs\Python\Python39\python.exe
> .\.venv39a\Scripts\activate
- Setup cvxpy in development mode:
> python setup.py develop
- Run
isort:
> isort .
- Run
pytestontest_problem.py:
> pytest .\cvxpy\tests\test_problem.py
Expected behavior
-
isortshould not change any files. - The test should pass.
Output
(.venv39a) PS D:\Dropbox\ariel-algorithms\cvxpy> pytest .\cvxpy\tests\test_problem.py
============================================================= test session starts =============================================================
platform win32 -- Python 3.10.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: D:\Dropbox\ariel-algorithms\cvxpy, configfile: pyproject.toml
collected 0 items / 1 error
=================================================================== ERRORS ====================================================================
________________________________________________ ERROR collecting cvxpy/tests/test_problem.py _________________________________________________
ImportError while importing test module 'D:\Dropbox\ariel-algorithms\cvxpy\cvxpy\tests\test_problem.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
C:\Users\user\AppData\Local\Programs\Python\Python310\lib\importlib\__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
cvxpy\__init__.py:26: in <module>
from cvxpy.problems.problem import Problem
cvxpy\problems\problem.py:35: in <module>
from cvxpy.reductions import InverseData
cvxpy\reductions\__init__.py:18: in <module>
from cvxpy.reductions.complex2real.complex2real import Complex2Real
cvxpy\reductions\complex2real\complex2real.py:23: in <module>
from cvxpy.reductions import InverseData, Solution
E ImportError: cannot import name 'InverseData' from partially initialized module 'cvxpy.reductions' (most likely due to a circular import) (D:\Dropbox\ariel-algorithms\cvxpy\cvxpy\reductions\__init__.py)
=========================================================== short test summary info ===========================================================
ERROR cvxpy/tests/test_problem.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================== 1 error in 0.62s ===============================================================
Version
- OS: Windows 10
- CVXPY Version: Current
@erelsgl The files you see differences in are exactly the known ones where the required ordering is conflicting with isort's rules.
That's why they are excluded in pyproject.toml:
[tool.isort]
include_trailing_comma = true
use_parentheses = true
extend_skip = [
"cvxpy/__init__.py",
"cvxpy/reductions/__init__.py",
"cvxpy/reductions/dcp2cone/atom_canonicalizers/__init__.py"
]
=> The issue seems to be that your isort config is not picking this up. I've just tried
pip install isort
isort .
on a fresh clone, and for me it does work. you can check
isort --show-config
In particular, make sure that directory is set to your cvxpy repo, and source points to the above-mentioned pyproject.toml file.
In isort --show-config I do see these skipped folders:
"extend_skip": [
"cvxpy/reductions/__init__.py",
"cvxpy/__init__.py",
"cvxpy/reductions/dcp2cone/atom_canonicalizers/__init__.py"
],
But still, isort . gives:
(.venv39cvxpy) PS D:\Dropbox\ariel-algorithms\cvxpy> isort .
Fixing D:\Dropbox\ariel-algorithms\cvxpy\cvxpy\__init__.py
Fixing D:\Dropbox\ariel-algorithms\cvxpy\cvxpy\reductions\__init__.py
Fixing D:\Dropbox\ariel-algorithms\cvxpy\cvxpy\reductions\dcp2cone\atom_canonicalizers\__init__.py
Skipped 5 files
@erelsgl From which directory are you running isort .?
Can you post the two values of the two keys I've mentioned above?
isortcfg.txt I am running isort from the main repository folder. The keys appear twice (see the attached file):
"directory": "",
...
"source": "defaults"
...
"source": "D:\\Dropbox\\ariel-algorithms\\cvxpy\\pyproject.toml"
...
"directory": "D:\\Dropbox\\ariel-algorithms\\cvxpy",
Do you have another isort install on your machine? Can you verify:
which isort
python -m isort .
You can also reach out via Discord, that might speed up debugging.
which isort does not work, but gcm isort gives:
CommandType Name Version Source
----------- ---- ------- ------
Application isort.exe 0.0.0.0 D:\tmp\.venv39cvxpy/Scripts\isort.exe
which is the folder where I installed the virtual environment.
python -m isort . gives the same result - it modifies these 3 files.
Is it possible that the problem is that the ignored files use "/" and the files on Windows use "\"?
@erelsgl That is possible, but I would be surprised if that was the cause. You can try changing it locally, though.
Can you alternatively try the following:
pip install pre-commit
pre-commit run -a
I changed the forward slashes to back slashes in extend_skip. This did not help, but after running isort . again, another thing happened: in many other files, a trailing comma was deleted. For example, in cvxpy/expressions/leaf.py, line 34 changed from:
from cvxpy.settings import (GENERAL_PROJECTION_TOL, PSD_NSD_PROJECTION_TOL,
SPARSE_PROJECTION_TOL,)
to
from cvxpy.settings import (GENERAL_PROJECTION_TOL, PSD_NSD_PROJECTION_TOL,
SPARSE_PROJECTION_TOL)
This does not cause any harm, but it is strange.
On the other hand, pre-commit run -a does work as expected: it skips the three files that it should, and it puts the comma back in.
@phschiele what do you think about closing this issue? To me, it seems like a bug in the usage / installation of isort. If we do anything, maybe it would just be to have this part of the web docs link to this issue so people know this might happen. If someone else has this problem then we can re-open.
@rileyjmurray Indeed seems like some issue with the local installation since the configuration clearly is being picked up for most users. Since the linters are usually considered towards the end of a PR, I don't think this would discourage someone from contributing, but I suppose adding a sentence to the docs wouldn't hurt. I can add this over the weekend.
I suggest to not tell new developer to run isort. Telling them to install pre-commit is enough as it will automatically run isort.
Adding some info that the pre-commit will run isort also will not hurt.
_
So, change https://github.com/cvxpy/cvxpy/blob/master/CONTRIBUTING.md#code-style to use pre-commit as default and run manually as the backup? #IMHO
I also just ran into this issue. Very weird. I just manually reverted changes. Here's the info on my isort install:
(cvx39) Rileys-MacBook-Pro:cvxpy riley$ which isort
/Users/riley/miniforge3/envs/cvx39/bin/isort
(cvx39) Rileys-MacBook-Pro:cvxpy riley$ isort -V
_ _
(_) ___ ___ _ __| |_
| |/ _/ / _ \/ '__ _/
| |\__ \/\_\/| | | |_
|_|\___/\___/\_/ \_/
isort your imports, so you don't have to.
VERSION 5.11.3
Here's my isort configuration for good measure.
(cvx39) Rileys-MacBook-Pro:cvxpy riley$ isort . --show-config
{
"_known_patterns": null,
"_section_comments": null,
"_section_comments_end": null,
"_skips": null,
"_skip_globs": null,
"_sorting_function": null,
"py_version": "py3",
"force_to_top": [],
"skip": [
".svn",
".direnv",
"build",
".tox",
".venv",
".mypy_cache",
".eggs",
"__pypackages__",
"node_modules",
".hg",
"venv",
".pants.d",
"_build",
".git",
".bzr",
"buck-out",
"dist",
".nox"
],
"extend_skip": [
"cvxpy/reductions/dcp2cone/atom_canonicalizers/__init__.py",
"cvxpy/__init__.py",
"cvxpy/reductions/__init__.py"
],
"skip_glob": [],
"extend_skip_glob": [],
"skip_gitignore": false,
"line_length": 79,
"wrap_length": 0,
"line_ending": "",
"sections": [
"FUTURE",
"STDLIB",
"THIRDPARTY",
"FIRSTPARTY",
"LOCALFOLDER"
],
"no_sections": false,
"known_future_library": [
"__future__"
],
"known_third_party": [],
"known_first_party": [],
"known_local_folder": [],
"known_standard_library": [
"rlcompleter",
"fractions",
"ssl",
"sys",
"trace",
"wave",
"resource",
"zipapp",
"math",
"telnetlib",
"test",
"signal",
"inspect",
"enum",
"argparse",
"traceback",
"imghdr",
"shelve",
"imp",
"pty",
"pstats",
"copy",
"ftplib",
"netrc",
"optparse",
"symtable",
"macpath",
"socket",
"platform",
"shlex",
"unittest",
"colorsys",
"select",
"codecs",
"dataclasses",
"asyncore",
"glob",
"keyword",
"encodings",
"calendar",
"builtins",
"http",
"datetime",
"random",
"distutils",
"fpectl",
"webbrowser",
"spwd",
"wsgiref",
"decimal",
"sre_compile",
"parser",
"threading",
"string",
"errno",
"msilib",
"lib2to3",
"cmd",
"io",
"smtpd",
"shutil",
"fcntl",
"xmlrpc",
"pwd",
"contextvars",
"curses",
"posixpath",
"pipes",
"tomllib",
"reprlib",
"logging",
"importlib",
"zipfile",
"base64",
"modulefinder",
"struct",
"zoneinfo",
"pprint",
"mailcap",
"ipaddress",
"pdb",
"locale",
"_ast",
"socketserver",
"fileinput",
"re",
"dummy_threading",
"imaplib",
"stat",
"mmap",
"sre",
"pydoc",
"graphlib",
"tkinter",
"timeit",
"weakref",
"syslog",
"sndhdr",
"formatter",
"chunk",
"hashlib",
"sre_constants",
"heapq",
"tokenize",
"_dummy_thread",
"html",
"urllib",
"grp",
"ossaudiodev",
"operator",
"sre_parse",
"turtledemo",
"bisect",
"hmac",
"symbol",
"selectors",
"pkgutil",
"gettext",
"subprocess",
"sunau",
"nis",
"venv",
"gc",
"ensurepip",
"dis",
"site",
"idlelib",
"functools",
"linecache",
"uuid",
"turtle",
"doctest",
"ast",
"types",
"msvcrt",
"difflib",
"pickletools",
"winsound",
"pickle",
"lzma",
"tempfile",
"poplib",
"audioop",
"code",
"typing",
"asynchat",
"xdrlib",
"codeop",
"statistics",
"faulthandler",
"binhex",
"bdb",
"fnmatch",
"pathlib",
"json",
"termios",
"contextlib",
"zlib",
"stringprep",
"zipimport",
"sqlite3",
"aifc",
"smtplib",
"tty",
"cmath",
"queue",
"secrets",
"_thread",
"winreg",
"tarfile",
"multiprocessing",
"atexit",
"itertools",
"abc",
"mailbox",
"cProfile",
"tracemalloc",
"cgi",
"posix",
"profile",
"quopri",
"sched",
"getpass",
"xml",
"token",
"time",
"asyncio",
"ntpath",
"gzip",
"collections",
"csv",
"pyclbr",
"unicodedata",
"runpy",
"mimetypes",
"dbm",
"array",
"uu",
"sysconfig",
"email",
"concurrent",
"binascii",
"ctypes",
"textwrap",
"copyreg",
"crypt",
"getopt",
"plistlib",
"compileall",
"os",
"configparser",
"cgitb",
"marshal",
"nntplib",
"numbers",
"bz2",
"readline",
"py_compile",
"tabnanny",
"filecmp",
"warnings"
],
"extra_standard_library": [],
"known_other": {},
"multi_line_output": "GRID",
"forced_separate": [],
"indent": " ",
"comment_prefix": " #",
"length_sort": false,
"length_sort_straight": false,
"length_sort_sections": [],
"add_imports": [],
"remove_imports": [],
"append_only": false,
"reverse_relative": false,
"force_single_line": false,
"single_line_exclusions": [],
"default_section": "THIRDPARTY",
"import_headings": {},
"import_footers": {},
"balanced_wrapping": false,
"use_parentheses": true,
"order_by_type": true,
"atomic": false,
"lines_before_imports": -1,
"lines_after_imports": -1,
"lines_between_sections": 1,
"lines_between_types": 0,
"combine_as_imports": false,
"combine_star": false,
"include_trailing_comma": true,
"from_first": false,
"verbose": false,
"quiet": false,
"force_adds": false,
"force_alphabetical_sort_within_sections": false,
"force_alphabetical_sort": false,
"force_grid_wrap": 0,
"force_sort_within_sections": false,
"lexicographical": false,
"group_by_package": false,
"ignore_whitespace": false,
"no_lines_before": [],
"no_inline_sort": false,
"ignore_comments": false,
"case_sensitive": false,
"sources": [
{
"py_version": "py3",
"force_to_top": [],
"skip": [
".svn",
".direnv",
"build",
".tox",
".venv",
".mypy_cache",
".eggs",
"__pypackages__",
"node_modules",
".hg",
"venv",
".pants.d",
"_build",
".git",
".bzr",
"buck-out",
"dist",
".nox"
],
"extend_skip": [],
"skip_glob": [],
"extend_skip_glob": [],
"skip_gitignore": false,
"line_length": 79,
"wrap_length": 0,
"line_ending": "",
"sections": [
"FUTURE",
"STDLIB",
"THIRDPARTY",
"FIRSTPARTY",
"LOCALFOLDER"
],
"no_sections": false,
"known_future_library": [
"__future__"
],
"known_third_party": [],
"known_first_party": [],
"known_local_folder": [],
"known_standard_library": [
"rlcompleter",
"fractions",
"ssl",
"sys",
"trace",
"wave",
"resource",
"zipapp",
"math",
"telnetlib",
"test",
"signal",
"inspect",
"enum",
"argparse",
"traceback",
"imghdr",
"shelve",
"imp",
"pty",
"pstats",
"copy",
"ftplib",
"netrc",
"optparse",
"symtable",
"macpath",
"socket",
"platform",
"shlex",
"unittest",
"colorsys",
"select",
"codecs",
"dataclasses",
"asyncore",
"glob",
"keyword",
"encodings",
"calendar",
"builtins",
"http",
"datetime",
"random",
"distutils",
"fpectl",
"webbrowser",
"spwd",
"wsgiref",
"decimal",
"sre_compile",
"parser",
"threading",
"string",
"errno",
"msilib",
"lib2to3",
"cmd",
"io",
"smtpd",
"shutil",
"fcntl",
"xmlrpc",
"pwd",
"contextvars",
"curses",
"posixpath",
"pipes",
"tomllib",
"reprlib",
"logging",
"importlib",
"zipfile",
"base64",
"modulefinder",
"struct",
"zoneinfo",
"pprint",
"mailcap",
"ipaddress",
"pdb",
"locale",
"_ast",
"socketserver",
"fileinput",
"re",
"dummy_threading",
"imaplib",
"stat",
"mmap",
"sre",
"pydoc",
"graphlib",
"tkinter",
"timeit",
"weakref",
"syslog",
"sndhdr",
"formatter",
"chunk",
"hashlib",
"sre_constants",
"heapq",
"tokenize",
"_dummy_thread",
"html",
"urllib",
"grp",
"ossaudiodev",
"operator",
"sre_parse",
"turtledemo",
"bisect",
"hmac",
"symbol",
"selectors",
"pkgutil",
"gettext",
"subprocess",
"sunau",
"nis",
"venv",
"gc",
"ensurepip",
"dis",
"site",
"idlelib",
"functools",
"linecache",
"uuid",
"turtle",
"doctest",
"ast",
"types",
"msvcrt",
"difflib",
"pickletools",
"winsound",
"pickle",
"lzma",
"tempfile",
"poplib",
"audioop",
"code",
"typing",
"asynchat",
"xdrlib",
"codeop",
"statistics",
"faulthandler",
"binhex",
"bdb",
"fnmatch",
"pathlib",
"json",
"termios",
"contextlib",
"zlib",
"stringprep",
"zipimport",
"sqlite3",
"aifc",
"smtplib",
"tty",
"cmath",
"queue",
"secrets",
"_thread",
"winreg",
"tarfile",
"multiprocessing",
"atexit",
"itertools",
"abc",
"mailbox",
"cProfile",
"tracemalloc",
"cgi",
"posix",
"profile",
"quopri",
"sched",
"getpass",
"xml",
"token",
"time",
"asyncio",
"ntpath",
"gzip",
"collections",
"csv",
"pyclbr",
"unicodedata",
"runpy",
"mimetypes",
"dbm",
"array",
"uu",
"sysconfig",
"email",
"concurrent",
"binascii",
"ctypes",
"textwrap",
"copyreg",
"crypt",
"getopt",
"plistlib",
"compileall",
"os",
"configparser",
"cgitb",
"marshal",
"nntplib",
"numbers",
"bz2",
"readline",
"py_compile",
"tabnanny",
"filecmp",
"warnings"
],
"extra_standard_library": [],
"known_other": {},
"multi_line_output": "GRID",
"forced_separate": [],
"indent": " ",
"comment_prefix": " #",
"length_sort": false,
"length_sort_straight": false,
"length_sort_sections": [],
"add_imports": [],
"remove_imports": [],
"append_only": false,
"reverse_relative": false,
"force_single_line": false,
"single_line_exclusions": [],
"default_section": "THIRDPARTY",
"import_headings": {},
"import_footers": {},
"balanced_wrapping": false,
"use_parentheses": false,
"order_by_type": true,
"atomic": false,
"lines_before_imports": -1,
"lines_after_imports": -1,
"lines_between_sections": 1,
"lines_between_types": 0,
"combine_as_imports": false,
"combine_star": false,
"include_trailing_comma": false,
"from_first": false,
"verbose": false,
"quiet": false,
"force_adds": false,
"force_alphabetical_sort_within_sections": false,
"force_alphabetical_sort": false,
"force_grid_wrap": 0,
"force_sort_within_sections": false,
"lexicographical": false,
"group_by_package": false,
"ignore_whitespace": false,
"no_lines_before": [],
"no_inline_sort": false,
"ignore_comments": false,
"case_sensitive": false,
"sources": [],
"virtual_env": "",
"conda_env": "",
"ensure_newline_before_comments": false,
"directory": "",
"profile": "",
"honor_noqa": false,
"src_paths": [],
"old_finders": false,
"remove_redundant_aliases": false,
"float_to_top": false,
"filter_files": false,
"formatter": "",
"formatting_function": null,
"color_output": false,
"treat_comments_as_code": [],
"treat_all_comments_as_code": false,
"supported_extensions": [
"pyi",
"pxd",
"pyx",
"py"
],
"blocked_extensions": [
"pex"
],
"constants": [],
"classes": [],
"variables": [],
"dedup_headings": false,
"only_sections": false,
"only_modified": false,
"combine_straight_imports": false,
"auto_identify_namespace_packages": true,
"namespace_packages": [],
"follow_links": true,
"indented_import_headings": true,
"honor_case_in_force_sorted_sections": false,
"sort_relative_in_force_sorted_sections": false,
"overwrite_in_place": false,
"reverse_sort": false,
"star_first": false,
"git_ls_files": {},
"format_error": "{error}: {message}",
"format_success": "{success}: {message}",
"sort_order": "natural",
"sort_reexports": false,
"split_on_trailing_comma": false,
"source": "defaults"
},
{
"include_trailing_comma": true,
"use_parentheses": true,
"extend_skip": [
"cvxpy/reductions/dcp2cone/atom_canonicalizers/__init__.py",
"cvxpy/__init__.py",
"cvxpy/reductions/__init__.py"
],
"source": "/Users/riley/Documents/cvxpydev/cvxpy/pyproject.toml"
}
],
"virtual_env": "",
"conda_env": "",
"ensure_newline_before_comments": false,
"directory": "/Users/riley/Documents/cvxpydev/cvxpy",
"profile": "",
"honor_noqa": false,
"src_paths": [
"/Users/riley/Documents/cvxpydev/cvxpy/src",
"/Users/riley/Documents/cvxpydev/cvxpy"
],
"old_finders": false,
"remove_redundant_aliases": false,
"float_to_top": false,
"filter_files": false,
"formatter": "",
"formatting_function": null,
"color_output": false,
"treat_comments_as_code": [],
"treat_all_comments_as_code": false,
"supported_extensions": [
"pyi",
"pxd",
"pyx",
"py"
],
"blocked_extensions": [
"pex"
],
"constants": [],
"classes": [],
"variables": [],
"dedup_headings": false,
"only_sections": false,
"only_modified": false,
"combine_straight_imports": false,
"auto_identify_namespace_packages": true,
"namespace_packages": [],
"follow_links": true,
"indented_import_headings": true,
"honor_case_in_force_sorted_sections": false,
"sort_relative_in_force_sorted_sections": false,
"overwrite_in_place": false,
"reverse_sort": false,
"star_first": false,
"git_ls_files": {},
"format_error": "{error}: {message}",
"format_success": "{success}: {message}",
"sort_order": "natural",
"sort_reexports": false,
"split_on_trailing_comma": false
}
@rileyjmurray I just set up a fresh codespace and only ran
pip install isort
isort .
and if worked, skipping 4 files. I tried this under bash and zsh.
I also checked that there were no relevant differences in our configs (unfortunately the sorting was different in some places, so it could not compare line by line).
Maybe you could post the results of isort . --show-files.
@phschiele I'm pretty sure I figured it out. Here's what happens when I run isort in the same directory as CVXPY's setup.py file:
(cvx39) Rileys-MacBook-Pro:cvxpy riley$ pwd
/Users/riley/Documents/cvxpydev/cvxpy
(cvx39) Rileys-MacBook-Pro:cvxpy riley$ isort cvxpy --check-only
Skipped 3 files
Meanwhile, here's what happens when I change directory to be level with cvxpy/__init__.py:
(cvx39) Rileys-MacBook-Pro:cvxpy riley$ cd cvxpy
(cvx39) Rileys-MacBook-Pro:cvxpy riley$ pwd
/Users/riley/Documents/cvxpydev/cvxpy/cvxpy
(cvx39) Rileys-MacBook-Pro:cvxpy riley$ isort . --check-only
ERROR: /Users/riley/Documents/cvxpydev/cvxpy/cvxpy/__init__.py Imports are incorrectly sorted and/or formatted.
ERROR: /Users/riley/Documents/cvxpydev/cvxpy/cvxpy/reductions/__init__.py Imports are incorrectly sorted and/or formatted.
ERROR: /Users/riley/Documents/cvxpydev/cvxpy/cvxpy/reductions/dcp2cone/atom_canonicalizers/__init__.py Imports are incorrectly sorted and/or formatted.
So it seems that the error can be attributed to isort seeing the configuration file based on the current working directory.