Support installing requirements from arbitrary keys in pyproject.toml

[dholth]

What's the problem this feature will solve?

It can sometimes be useful to install from a pyproject.toml that has a list of install_requires = [] without having to build the whole package.

pip install -r pyproject.toml --key tool.enscons.install_requires would open pyproject.toml, look up ['tool']['enscons']['install_requires'], and install as if those items had been passed as arguments to pip install, or written out to a requirements.txt file and then installed.

It wouldn't work correctly if the package referenced its own extras, unless a second key for the extras dict was passed.

If this feature was added it would be easy to support .json as well.

[pfmoore]

I'd rather support -r -, which would then allow you to use some sort of TOML equivalent of jq¹ to read the values you want out and pass them in that way. That would be more general than a specific pyproject.toml reading feature.

¹ A quick search found https://github.com/jamesmunns/tomlq

[uranusjr]

Personally I tend to think all -r usages on “non-locked” requirement specifications should be discouraged, since it does not fit well into the recent best practice trend of lock files. I would be in favour of introducing a new requirements file format, but pyproject.toml should not be it; the lock file format should.

[pradyunsg]

I'd rather support -r -

https://github.com/pypa/pip/issues/7822

[chrisjbillington]

I'm looking for something like this too.

My use case is making sdists in CI. pip wheel with its PEP 518 support means I no longer need to install build dependencies (from a requirements.txt or otherwise) prior to making a wheel, and I don't need to play import-guarding tricks in setup.py to make sure it is runnable without build requirements.

That's great if I'm making a wheel, but the same problems exist for sdists, which remain unsolved. Even though I'm not building anything for an sdist, some of my projects still import cython in their setup.py, they still use setuptools_scm for versioning. So to make an sdist some of these requirements are needed, and I'd have to resort to tricks again to delay importing the others, or install all build dependencies before running python setup.py sdist.

So what I would really like is a pip sdist command that does the build isolation process and installs the 'build' dependencies but makes an sdist instead of a wheel. But a pip install -r pyproject.toml would work pretty well too - no isolation but it doesn't matter on the CI server.

pip install -r pyproject.toml is a bit nicer than

pip install toml && python -c 'import toml; c = toml.load("pyproject.toml")
print("\n".join(c["build-system"]["requires"]))' | pip install -r /dev/stdin

Which is what I'm currently considering putting in my CI.

[sbidoul]

@chrisjbillington for building sdists, you could try python -m pep517.build from the pep517 project. Maybe one day pip or twine or some other tool will gain that capability, but in the meantime that should do the job.

[chrisjbillington]

Thanks @sbidoul, looks like I should use pep517 until the discussion about where the 'build an sdist according to PEP 517' tool belongs is settled.

Maybe pep517 should be renamed pyproject and make an official entry point to python -m pep517.build, as pyproject build

[uranusjr]

(Alas, pyproject is taken on PyPI.)

[chrisjbillington]

(Alas, pyproject is taken on PyPI.)

Ah, unluckly. Even if the author were happy to relinquish it (it doesn't look active, though the author's other projects are), it'd be strange to re-use a package name for something completely different.

An alternative could be to include a renamed pep517 it in the standard library, or ship it with pip. Then it wouldn't need a PyPI name.

[pradyunsg]

(Alas, pyproject is taken on PyPI.)

That's definitely an invalid project. Filed https://github.com/pypa/pypi-support/issues/417.

[chrisjbillington]

It looks like a real project to me, and seems to predate the PEPs introducing pyproject.toml so doesn't look like intentional name-squatting. It's just unmaintained.

[pandichef]

This works pretty well FWIW... pyproject.toml:

[install_requires]
django = "*"

[extras_require.dev]
django-debug-toolbar = "*"

[build-system]
requires = ["setuptools", "wheel", "toml"]

setup.py:

from os.path import dirname, abspath, join
from setuptools import setup, find_packages
import toml

with open("pyproject.toml", "r") as f:
   requirements = toml.loads(f.read())

prod = requirements['install_requires']
dev = requirements['extras_require']['dev']

setup(
    install_requires=[x + prod[x] if prod[x] != "*" else x for x in prod],
    extras_require={'dev': [x + dev[x] if dev[x] != "*" else x for x in dev]},
)

Then just pip install -e .[dev] as usual. I landed on this solution after losing patience with the slowness of pipenv.

[uranusjr]

Please do not use non-standard top-level keys in pyproject.toml. All top-level keys except tool are reserved for future use by Python packaging, according to PEP 518.

[cassamajor]

I use flit as the build-system in my pyproject.toml.

Using a minimal example, my pyproject.toml:

[build-system]
requires = ["flit_core >=2,<4"]
build-backend = "flit_core.buildapi"

[tool.flit.metadata]
module = "foobar"
author = "Sir Robin"
author-email = "[email protected]"
home-page = "https://github.com/sirrobin/foobar"

and foobar.py:

""" Hello """

__version__ = '1'

I can use pip install . or pip install foobar to install the project

(venv) $ pip install .
Processing /user/directory
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: foobar
  Building wheel for foobar (PEP 517) ... done
  Created wheel for foobar: filename=foobar-1-py2.py3-none-any.whl size=947 sha256=..snip..
  Stored in directory: ...snip..
Successfully built foobar
Installing collected packages: foobar
Successfully installed foobar-1

If you specify requires under the [tool.flit.metadata] stanza, then pip will also install the specified requirements.

Does this solve the problem that is faced by @dholth and @chrisjbillington?

[chrisjbillington]

My problem is completely addressed by python -m pep517.build -s ., which other than the awkward name of the pep517 project has no drawbacks that I've encountered. I'm not in need of any other solutions to this problem.

[sbidoul]

@chrisjbillington in the meantime, build became a thing so the awkward name issue is resolved too.

[uranusjr]

Or, if you like things from the other direction, build also has an extra awkward endpoint pyproject-build 😛

[casperdcl]

slight tweak to https://github.com/pypa/pip/issues/8049#issuecomment-633845028:

pip install toml
python3 -c \
  'import toml; c=toml.load("pyproject.toml"); print("\0".join(c["build-system"]["requires"]), end="")' \
  | xargs -0 pip install

though I agree that if pip sdist becomes a thing (similar to pip wheel --no-deps . replacing python setup.py bdist_wheel) then this hack isn't required... Is there an issue requesting pip sdist buried somewhere?

[flying-sheep]

I wouldn’t call it a hack, it’s simply very manual. A hack is something reaching into APIs you’re not supposed to muck around with. This uses bog-standard, spec compliant ways to do what we want.

But it is awkward. Slightly less is e.g. using rq to convert it to JSON and then jq to query and transform:

rq -t <pyproject.toml | jq -r '.["build-system"]["requires"][]' | xargs -d '\n' pip install

PS: pip install has no -y flag last time I checked.

[casperdcl]

Sure, pip install yq :)

[dudil]

@pandichef - I'm trying to utilize docker images as a replacement to the virutalenv hell and pipenv becoming something obsolete. Your comment is surprising as I never thought of using the packaging mechanism for local installation and this is an interesting approach for what I'm looking for - so thank you!

Did you look into @Fauxsys suggestion? To me it looks similar to what you were trying to achieve, do you think the same? Asking since I'm not 100% they are indeed trying to solve the same issue.

[pandichef]

@dudil It looks like Fauxsys is using flit, which I've never used. My approach was born of frustration with the existing package management tools.

• "pip install -r requirements.txt" doesn't seem to produce consistent dependencies (which is the raison d'etre of pipenv)
• pipenv is soooooo slow
• poetry seems nice, but it does a lot and I don't like big opinionated stuff in general; also, I don't like relying on some 3rd party team to deliver features, fix bugs, etc., especially in the wake of the pipenv fiasco

My approach has four nice features:

• It produces consistent results AFAICT (see below)
• It's fast
• It's doesn't rely on some opinionated third-party library like flit or poetry
• It allows you to create a new pypi package with almost no overhead

Note: My approach might not be consistent in all cases. (I haven't really stress-tested it in any way.) If you really need rock-solid consistency, then pipenv or poetry is the only way to go.

[dudil]

@pandichef Totally agree with you on the points mentioned. I will also mention that poetry is really a mess of opinionated tools, and I don't like this opinion. Most notably, and it was mentioned here as well is the "(Monty) Python and the holy grail of lock files". People are killing the echo system with that concept but not relevant here.

Anyway, flit is more simple and from reading PEP-517 it is pretty much created for flit. Since your template does look like something needs to be reused I thought might be using flit is doing similar but packing that for reuse.

[flying-sheep]

Let’s talk about use cases. I can only come up with one, but it’s a very useful one:

Cutting off the root of a dependency tree for granular building.

Build systems like Make or Docker work by specifying dependencies on files, and when you change these files, the steps depending on them have to be redone.

So to save time (and in Docker’s case, layer uploading bandwidth), you do this:

1. pull in / depend on the requirements file
2. install the requirements (usually needs internet access)
3. pull in / depend on the code
4. install the package you’re developing

That means step 1 and 2 only has to be redone when the requirements change. All other code changes only affect the very fast steps 3 and 4.

[uranusjr]

Hi, I see the recent messages have drifted away from the original topic. May I request people to move the conversation to a different venue? I understand the topic is important and deserves a discussion, but it does injustice to both the original feature request (make pip support installing from pyproject.toml) and the topic recent messages focus on (workflow to build pyproject.toml projects without tools that produce them) in one thread. They are both very important, and deserve their own threads, e.g. for future reference.

[flying-sheep]

That’s exactly what I was getting at: Why have that feature?

The only reason I can come up with is the above. Therefore (unless there’s more use cases) we can also design a feature that serves that use case and looks differently.

E.g. pip install --only-deps . or so.

[dudil]

@uranusjr

You are right on the point mentioned, appreciate the focus on the original request. I will add to the original thread, and hope you will find it useful (my opinion only - not absolute truth):

1. Having the ability to install requirements directly from pyproject.toml should had been an implemented as part of PEP-517
2. I pretty much believe that the fact both @pandichef and @Fauxsys presented a real life working "hack" means that this requirement is de facto required and its lack is posing difficulties on the community.
3. The OP smartly enough didn't mentioned any lock file mechanism. I believe this it a smart decision. Trying to fix two different issues at once might not be a good strategy in that case. @uranusjr - Since you were the first to mention it here, maybe you agree this deserve a separate (very much important) feature request?

I will be happy to provide a PR implementing this request if ok.

[uranusjr]

We are already working on a lock file solution. I don’t think there’s a tracking issue yet, everything is internal at this point. The lock file format is very subjective and we don’t want to attract public input at this point since it would make the conversation diverge too much.

The current short-term goal is to publish a PEP on the topic and field conversation there. We’ll post to https://discuss.python.org/ when a PEP is drafted (as required by the process), so make sure to keep an eye there is you’re interested in this.

[damarvin]

(Keeping aside the lock file part of the discussion—my understanding is, this is generated and rather "throw-away")

Trying to push this forward, I mean the issue is, that PEP 621 (in Final state) allows (from my point of view unnecessarily) different keys for the same things https://www.python.org/dev/peps/pep-0621/#dependencies-optional-dependencies

From all these pip install -r pyproject.toml [--optional] would need to read and work like you would have requirements.txt requirements-dev.txt ...and sure, latter 2 files should not be there (maintained in a project in parallel), when there is a pyproject.toml.

Note, there is the trend to use pyproject.toml files even w/o the package-build parts (flit | poetry | setuptools), e.g. pytest, black: So pip might want to define a further variant or prefer/promote 1 of the 3 like

[tool.poetry.dependencies]
# ...
[tool.poetry.dev-dependencies]
# ...

[uranusjr]

PEP 621 is for storing dependencies of a project publishing a Python distribution (i.e. something on PyPI). It is expected to be read by a build back-end, e.g. setuptools, flit (more specifically flit-core), poetry (more specifically poetry-core). Pip is not a build backend.

To use PEP 621 in pip, you should build your project as a Python distribution (using a back-end that supports PEP 621; I don't know how various back-ends are doing), and pip install . in the project root.

[dholth]

I saw a recent Docker pattern where requirements.txt was installed, and then the application was pip installed in another layer. That improves caching if the application is updated more frequently than its requirements.