flit
flit copied to clipboard
Published
20 hours ago •
pypa
pypa
Editable build could be broken when user miss docstring for description
I had a weird behavior which one of the two packages were working fine for editable installation, but the other one was failed to install emitting No matching distribution found
After digging a bit, I found I had a mistake not to write docstring with dynamic description.
From the code (https://github.com/pypa/flit/blob/main/flit_core/flit_core/buildapi.py#L34), when user missed the required docstring, flit returns the package which will be checked as build dependency. I'm using pip for frontend and pip seems to consider the build dependencies to be installed (https://github.com/pypa/pip/blob/main/src/pip/_internal/distributions/sdist.py#L117) with ignore_installed=True flag as build dependencies should (or may?) be installed from the clean environment.
It results in distribution discovery failure when user use dev branch without publishing yet (e.g. 0.0.1dev).
I'm not sure whether it's front-end issue or flit side, but it seems to be more related to flit side as I thought it's OK to set ignore_installed=true for build dependencies.
As I took a lot of time to figure it out so I made this issue to hope someone who will have same mistake like me to get a viable hint, and please let me know if flit is not the place to solve the issue.
Thanks.
Sorry, I'm a bit confused. Could you describe exactly what you're trying and what error you see?
I've just tried removing the docstring from a package and then doing pip install --editable . with the latest pip, and while it's rather verbose, it does include the relevant error message from Flit:
$ pip install --editable .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/takluyver/Code/flit/flit_core/flit_core/tests/samples/pep621
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Installing backend dependencies ... done
Preparing editable metadata (pyproject.toml) ... error
error: subprocess-exited-with-error
× Preparing editable metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [14 lines of output]
Traceback (most recent call last):
File "/home/takluyver/.local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
main()
File "/home/takluyver/.local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/takluyver/.local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 191, in prepare_metadata_for_build_editable
return hook(metadata_directory, config_settings)
File "/tmp/pip-build-env-7drhdtkz/overlay/lib/python3.10/site-packages/flit_core/buildapi.py", line 49, in prepare_metadata_for_build_wheel
metadata = make_metadata(module, ini_info)
File "/tmp/pip-build-env-7drhdtkz/overlay/lib/python3.10/site-packages/flit_core/common.py", line 408, in make_metadata
md_dict.update(get_info_from_module(module, ini_info.dynamic_metadata))
File "/tmp/pip-build-env-7drhdtkz/overlay/lib/python3.10/site-packages/flit_core/common.py", line 211, in get_info_from_module
raise NoDocstringError(
flit_core.common.NoDocstringError: Flit cannot package module without docstring, or empty docstring. Please add a docstring to your module (/home/takluyver/Code/flit/flit_core/flit_core/tests/samples/pep621/module1a.py).
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Sorry I was not clear enough. I myself was confused to understand my description after a few days. :joy:
Here's how to reproduce
(.venv) issue534$ tree
.
├── bar
│ ├── bar
│ │ ├── __init__.py
│ └── pyproject.toml
└── foo
├── foo
│ ├── __init__.py
└── pyproject.toml
Actually there were two packages involved. I confirmed that just one package worked fine emitting your log
Package foo has valid doc string, and installed correctly.
(.venv) issue534$ cat foo/pyproject.toml
[build-system]
requires = ["flit_core >=3.2,<4"]
build-backend = "flit_core.buildapi"
[project]
name = "foo"
version = "0.1.0.dev0"
dynamic = ["description"]
(.venv) issue534$ cat foo/foo/__init__.py
"""foo"""
But the problem is the other package bar. It depends on foo package and no docstring there. (This was my mistake)
(.venv) issue534$ cat bar/pyproject.toml
[build-system]
requires = ["flit_core >=3.2,<4"]
build-backend = "flit_core.buildapi"
[project]
name = "bar"
version = "0.1.0.dev0"
dynamic = ["description"]
dependencies = ["foo == 0.1.*"]
(.venv) issue534$ cat bar/bar/__init__.py
(.venv) issue534$
If I try to install bar with editable flag after foo installed, it failed like below
(.venv) bar$ pip list
Package Version Editable project location
---------- ---------- -----------------------------------------------
foo 0.1.0.dev0 /home/ys/Practice/languages/python/issue534/foo
pip 22.0.4
setuptools 58.1.0
(.venv) bar$ pip install -e .
Obtaining file:///home/ys/Practice/languages/python/issue534/bar
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Installing backend dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 1
╰─> [2 lines of output]
ERROR: Could not find a version that satisfies the requirement foo==0.1.* (from versions: none)
ERROR: No matching distribution found for foo==0.1.*
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
(.venv) bar$
If I add the missing docstring to bar, I managed to install the package then.
(.venv) bar$ cat bar/__init__.py
"""bar"""
(.venv) bar$ pip install -e .
Obtaining file:///home/ys/Practice/languages/python/issue534/bar
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: foo==0.1.* in /home/ys/Practice/languages/python/issue534/.venv/lib/python3.10/site-packages (from bar==0.1.0.dev0) (0.1.0.dev0)
Building wheels for collected packages: bar
Building editable for bar (pyproject.toml) ... done
Created wheel for bar: filename=bar-0.1.0.dev0-py2.py3-none-any.whl size=937 sha256=3571d18954142a35b0760ff330019a5d384b4590b5067a9a1a41074745c0fee4
Stored in directory: /tmp/pip-ephem-wheel-cache-k7zor9xj/wheels/86/88/6e/96d01046ed08574cdea413fa27de12cb2dfe4780faea4805bf
Successfully built bar
Installing collected packages: bar
Successfully installed bar-0.1.0.dev0
Can you run the failing pip install run with -vvv and post the output of that?
Sorry for too late response. I did not recognize @pradyunsg
Here's verbose log for failure
(.venv) bar$ pip install -e .
Obtaining file:///home/ys/Source/Practice/languages/python/issue534/bar
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Installing backend dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 1
╰─> [4 lines of output]
ERROR: Could not find a version that satisfies the requirement foo==0.1.* (from versions: none)
ERROR: No matching distribution found for foo==0.1.*
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/home/ys/Source/Practice/languages/python/issue534/.venv/bin/python -m pip install --upgrade pip' command.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/home/ys/Source/Practice/languages/python/issue534/.venv/bin/python -m pip install --upgrade pip' command.
(.venv) bar$
(.venv) bar$ pip install -e . -vvv
Using pip 22.0.4 from /home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip (python 3.9)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-ephem-wheel-cache-d3ak3__e
Created temporary directory: /tmp/pip-req-tracker-cmvalpf9
Initialized build tracking at /tmp/pip-req-tracker-cmvalpf9
Created build tracker: /tmp/pip-req-tracker-cmvalpf9
Entered build tracker: /tmp/pip-req-tracker-cmvalpf9
Created temporary directory: /tmp/pip-install-kkffeywm
Obtaining file:///home/ys/Source/Practice/languages/python/issue534/bar
Added file:///home/ys/Source/Practice/languages/python/issue534/bar to build tracker '/tmp/pip-req-tracker-cmvalpf9'
Created temporary directory: /tmp/pip-build-env-q5r_5__7
Created temporary directory: /tmp/pip-standalone-pip-mxs4yfbf
Running command pip subprocess to install build dependencies
Using pip 22.0.4 from /tmp/pip-standalone-pip-mxs4yfbf/__env_pip__.zip/pip (python 3.9)
Collecting flit_core<4,>=3.2
Using cached flit_core-3.8.0-py3-none-any.whl (62 kB)
Installing collected packages: flit_core
Successfully installed flit_core-3.8.0
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/home/ys/Source/Practice/languages/python/issue534/.venv/bin/python -m pip install --upgrade pip' command.
Installing build dependencies ... done
Running command Checking if build backend supports build_editable
Checking if build backend supports build_editable ... done
Running command Getting requirements to build editable
Getting requirements to build editable ... done
Created temporary directory: /tmp/pip-standalone-pip-9l28clcb
Running command pip subprocess to install backend dependencies
Using pip 22.0.4 from /tmp/pip-standalone-pip-9l28clcb/__env_pip__.zip/pip (python 3.9)
ERROR: Could not find a version that satisfies the requirement foo==0.1.* (from versions: none)
ERROR: No matching distribution found for foo==0.1.*
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/home/ys/Source/Practice/languages/python/issue534/.venv/bin/python -m pip install --upgrade pip' command.
error: subprocess-exited-with-error
× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /home/ys/Source/Practice/languages/python/issue534/.venv/bin/python /tmp/pip-standalone-pip-9l28clcb/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-q5r_5__7/normal --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'foo == 0.1.*'
cwd: [inherit]
Installing backend dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Exception information:
Traceback (most recent call last):
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
status = run_func(*args)
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
return func(self, options, args)
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 339, in run
requirement_set = resolver.resolve(
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 75, in resolve
collected = self.factory.collect_root_requirements(root_reqs)
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 506, in collect_root_requirements
req = self._make_requirement_from_install_req(
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 468, in _make_requirement_from_install_req
cand = self._make_candidate_from_link(
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 188, in _make_candidate_from_link
self._editable_candidate_cache[link] = EditableCandidate(
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 313, in __init__
super().__init__(
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__
self.dist = self._prepare()
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 227, in _prepare
dist = self._prepare_distribution()
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 323, in _prepare_distribution
return self._factory.preparer.prepare_editable_requirement(self._ireq)
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 609, in prepare_editable_requirement
dist = _get_prepared_distribution(
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 58, in _get_prepared_distribution
abstract_dist.prepare_distribution_metadata(finder, build_isolation)
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/distributions/sdist.py", line 45, in prepare_distribution_metadata
self._install_build_reqs(finder)
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/distributions/sdist.py", line 108, in _install_build_reqs
self.req.build_env.install_requirements(
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/build_env.py", line 202, in install_requirements
self._install_requirements(
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/build_env.py", line 260, in _install_requirements
call_subprocess(
File "/home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip/_internal/utils/subprocess.py", line 224, in call_subprocess
raise error
pip._internal.exceptions.InstallationSubprocessError: pip subprocess to install backend dependencies exited with 1
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/home/ys/Source/Practice/languages/python/issue534/.venv/bin/python -m pip install --upgrade pip' command.
Removed file:///home/ys/Source/Practice/languages/python/issue534/bar from build tracker '/tmp/pip-req-tracker-cmvalpf9'
Removed build tracker: '/tmp/pip-req-tracker-cmvalpf9'
(.venv) bar$ vi bar/__init__.py
After adding docstring,
(.venv) bar$ pip install -e . -vvv
Using pip 22.0.4 from /home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages/pip (python 3.9)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-ephem-wheel-cache-axr297gn
Created temporary directory: /tmp/pip-req-tracker-xkr7xz8v
Initialized build tracking at /tmp/pip-req-tracker-xkr7xz8v
Created build tracker: /tmp/pip-req-tracker-xkr7xz8v
Entered build tracker: /tmp/pip-req-tracker-xkr7xz8v
Created temporary directory: /tmp/pip-install-buk43w7s
Obtaining file:///home/ys/Source/Practice/languages/python/issue534/bar
Added file:///home/ys/Source/Practice/languages/python/issue534/bar to build tracker '/tmp/pip-req-tracker-xkr7xz8v'
Created temporary directory: /tmp/pip-build-env-chnpzrnq
Created temporary directory: /tmp/pip-standalone-pip-hwmh6ahi
Running command pip subprocess to install build dependencies
Using pip 22.0.4 from /tmp/pip-standalone-pip-hwmh6ahi/__env_pip__.zip/pip (python 3.9)
Collecting flit_core<4,>=3.2
Using cached flit_core-3.8.0-py3-none-any.whl (62 kB)
Installing collected packages: flit_core
Successfully installed flit_core-3.8.0
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/home/ys/Source/Practice/languages/python/issue534/.venv/bin/python -m pip install --upgrade pip' command.
Installing build dependencies ... done
Running command Checking if build backend supports build_editable
Checking if build backend supports build_editable ... done
Running command Getting requirements to build editable
Getting requirements to build editable ... done
Created temporary directory: /tmp/pip-modern-metadata-6rneano9
Running command Preparing editable metadata (pyproject.toml)
Preparing editable metadata (pyproject.toml) ... done
Source in /home/ys/Source/Practice/languages/python/issue534/bar has version 0.1.0.dev0, which satisfies requirement bar==0.1.0.dev0 from file:///home/ys/Source/Practice/languages/python/issue534/bar
Removed bar==0.1.0.dev0 from file:///home/ys/Source/Practice/languages/python/issue534/bar from build tracker '/tmp/pip-req-tracker-xkr7xz8v'
Requirement already satisfied: foo==0.1.* in /home/ys/Source/Practice/languages/python/issue534/.venv/lib/python3.9/site-packages (from bar==0.1.0.dev0) (0.1.0.dev0)
Created temporary directory: /tmp/pip-unpack-zpc7ihtt
Building wheels for collected packages: bar
Created temporary directory: /tmp/pip-wheel-dp4idaus
Destination directory: /tmp/pip-wheel-dp4idaus
Running command Building editable for bar (pyproject.toml)
Building editable for bar (pyproject.toml) ... done
Created wheel for bar: filename=bar-0.1.0.dev0-py2.py3-none-any.whl size=945 sha256=95edbb8c5ea7f41367c850e2cb38817f1cc8fbee0182156add07a78dc64fd2fd
Stored in directory: /tmp/pip-ephem-wheel-cache-axr297gn/wheels/aa/e2/a6/6e6128a1a5ff0f61c0869a6a92f6594ef38861ed4e5dcfa201
Successfully built bar
Installing collected packages: bar
Successfully installed bar-0.1.0.dev0
WARNING: You are using pip version 22.0.4; however, version 23.1.2 is available.
You should consider upgrading via the '/home/ys/Source/Practice/languages/python/issue534/.venv/bin/python -m pip install --upgrade pip' command.
Removed build tracker: '/tmp/pip-req-tracker-xkr7xz8v'
I think I understand, and this is more or less expected, albeit not ideal.
Flit tries to get the version number & docstring by static analysis first - i.e. parsing the code but not executing it. If that doesn't work, it falls back to importing your module, i.e. executing its top-level code. If it's trying that, any dependencies of your package become build dependencies, because they may need to be installed to import it. Pip builds your package in an isolated environment by default, so although foo is already installed in your target environment, it's not installed in the build environment, and pip can't find it to install.
I think using flit install --symlink or --pth-file instead of pip install -e . will likely fail with a clearer error message.