pypa
PEP 723 support
What's the problem this feature will solve?
This feature request represents support for installing the requirements of a single-file script with inline metadata (PEP 723) using pip.
Pessimistically assuming that this feature request is rejected, this issue is also being filed as a home for a specific record of the maintainer rationale for not including this functionality in pip.
Describe the solution you'd like
I am imagining functionality that would be similar to the existing --requirement argument, with a suitably distinct name to indicate that the metadata format is the one defined by PEP 723 and the living PyPA spec.
Perhaps something like:
$ cat standalone_script.py
# /// script
# requires-python = ">=3.11"
# dependencies = [
# "requests<3",
# "rich",
# ]
# ///
import requests
from rich.pretty import pprint
resp = requests.get("https://peps.python.org/api/peps.json")
data = resp.json()
pprint([(k, v["title"]) for k, v in data.items()][:10])
$ pip install --script standalone_script.py
...
Where the pip install invocation is equivalent to pip install "requests<3" "rich" if the requires-python bound is satisfied
Alternative Solutions
I know that other tools¹ in the Python packaging ecosystem do support PEP 723, but these tools are not "included batteries" nearly as often as pip is. In my estimation, supporting in-line metadata in pip will improve the experience of many users (especially non-programmers) who just want to run a script they've been given and are starting with a "typical" bare Python environment.
¹non-exhaustive list: pipx, hatch, pip-run
Additional context
The reference implementation for parsing inline metadata seems to be enough to retrieve the list of requirements from a file. I imagine that the result of this can be plumbed into existing machinery used for satisfying dependencies listed in requirements files.
It's worth mentioning that there is a potential argument for allowing multiple scripts to be parsed and installed from in a single pip invocation (in the same way that --requirement can be given multiple times). I think it makes sense but I can see why there might be some reluctance about doing it since it is more or less directly against the grain of the primary "single script" use-case of inline metadata.
Code of Conduct
- [X] I agree to follow the PSF Code of Conduct.
I feel that this is a reasonable request, although personally I consider it to be a suboptimal use of inline script metadata - the point of that data is for a tool that runs the script to transparently manage the environment the script runs in. Manually managing the environment with pip is not the intended use and I wouldn’t want to see it become a common pattern.
Having said that, if someone wants to submit a PR, I wouldn’t object. It’s not a feature I intend to work on myself, though.
I feel that this is a reasonable request, although personally I consider it to be a suboptimal use of inline script metadata - the point of that data is for a tool that runs the script to transparently manage the environment the script runs in. Manually managing the environment with pip is not the intended use and I wouldn’t want to see it become a common pattern.
Having said that, if someone wants to submit a PR, I wouldn’t object. It’s not a feature I intend to work on myself, though.
I'm willing to dabble into this. Would it be possible to just shift the solution from pipx? If so, a pull request should be quite easy.
I'm not sure what part of the pipx code you're thinking of. Parsing the PEP 723 metadata uses the code from the PEP itself, I believe, so it'll be the same code wherever you take it from. Most of the rest of the pipx code is about managing virtual environments, which is not suitable for pip.
The "fun" bit of this would likely be around designing the UI. Would the new --script-requirements (or whatever you call it) argument be allowed with --requirements, or are they mutually exclusive? Should --constraints be allowed? What do we do about a requires-python in the script requirements? Fail with an error if the target environment uses a Python version that doesn't match the constraint? How would that work with --target, where we don't know what Python version will be used?
I don't want to over-complicate the problem here, but I'd strongly encourage you to think about the design, rather than hoping to just copy/paste some code from another project. A bit of test-driven design, where you start by writing some tests to exercise the behaviour you want, and then write code to implement it (and hence "fix" the tests) would be very useful here, IMO.
Thinking about this issue again. I agree with Paul that the venv part of this is not pip's problem, thought I'd rattle off my personal answers to the explicit (but non-exhaustive) questions:
Would the new --script-requirements (or whatever you call it) argument be allowed with --requirements, or are they mutually exclusive? Should --constraints be allowed?
I don't really see any reason to exclude --requirements or --constraints, but I would want to avoid using the word "requirements" in the new functionality to avoid conflating the two distinct formats. I'm not heavily attached to the --script argument in the initial filing.
What do we do about a requires-python in the script requirements? Fail with an error if the target environment uses a Python version that doesn't match the constraint?
That's the way I read the PEP. It's not quite a requirement, but it does say that this SHOULD happen. I can't think of any reason to go against the recommendation, aside from…
How would that work with --target, where we don't know what Python version will be used?
That's an interesting question. I can see a case for ignoring requires-python in this scenario and issuing a warning if the Python running pip is known to not be compatible.
Would it be possible to just https://github.com/pypa/pipx/pull/1100 the solution from pipx? If so, a pull request should be quite easy.
I'd strongly encourage you to think about the design, rather than hoping to just copy/paste some code from another project
+1, I think implementing this is going to require the kind of motivated design from where pip currently is that Paul is talking about. There's just too much difference with pipx in terms of what the tools do. In particular, pip is not and IMO should not become a script runner.
Making a note here that the implications of this feature for pip download and pip wheel (the other commands that accept --requirements) should also be considered when drafting a design for this feature. I am personally most interested in the behavior for pip install but it makes sense to track the capabilities of --requirements as closely as is practical.
Edit: from a minimal draft I spent the last few hours putting together, it looks like functionality for those commands is "free" inasmuch as the commands that build requirement sets all get this functionality via req_command.py
A use case for this which I've come across is AWS lambdas where you want to install your dependencies to a specific folder so that you can then zip it up with your script. Allowing pip to read the dependencies from the inline script metadata means you don't need an extra requirements file, and that you can separate your runtime dependencies from your development dependencies.
@clbarnes not to poo on the idea... but worth noting that a lot of people prefer read-only containers which wouldn't be possible with that method... unless there were a corresponding install method that doesn't execute the script after checking/installing/caching dependencies.
I see 2 paths where pip could do this
pip run-script myscript.py use the hode for build venvs to make the script env and dispose of it afterwards
Anyone who wants cache needs another tool
pip install --script-deps myscript threading the script like a requirement file
Id argue against providing a way to install the script itself as the inline metadata is intented to run it directly
unless there were a corresponding install method that doesn't execute the script after checking/installing/caching dependencies.
Yes, I was thinking along the lines of using the script dependencies like a requirements file, so that you could bootstrap an environment which could run the script. So you'd do e.g. pip install -r myscript.py and it would look for inline dependencies.
pip run-script myscript.pyuse the hode for build venvs to make the script env and dispose of it afterwards
There was a big discussion on having a pip run command back in https://github.com/pypa/pip/issues/3971. I was ambivalent at the time, because I like the functionality but I don't think environment management is a natural thing to include in pip. Ultimately that issue didn't get implemented because we couldn't get consensus on the approach or desirability of the feature.
Given that pipx run and uv run satisfy this need, I don't think it's something that we should add to pip. If people want something that's in the stdlib, I'd suggest they lobby the core developers directly. A stdlib solution doesn't need to be in pip, and arguably would be better separate - including it in the runpy module would seem more natural, or maybe even option to the interpreter itself to run a script with dependencies directly.
pip install --script-deps myscriptthreading the script like a requirement file
As I said above, I don't object to this (although I'm not supportive of it either). I don't really see the value - for me script metadata is useful to allow running a script with dependencies without needing to manually manage environments - but if someone wants to design the feature and write a PR, I'm not going to reject it. But I'm also not going to spend time on it myself - my time is limited these days, and this isn't something I'd prioritise.
A stopgap: tagging this onto the end of the reference implementation for reading script metadata
# myscript.py
from pathlib import Path
import sys
scripts = sys.argv[1:]
for p in scripts:
# `read` is the function defined in the reference implementation
# https://packaging.python.org/en/latest/specifications/inline-script-metadata/#reference-implementation
d = read(Path(p).read_text())
if d is None:
continue
deps: list[str] = d.get("dependencies", [])
for dep in deps:
print("".join(dep.split()))
and then doing
myscript.py script_with_metadata.py | pip install -r /dev/stdin
Yeah, this is one reason I don't think this is worth adding to pip - it's so easy to (write a little script to) extract the dependencies from the script data, and then you can feed them into a normal -r option or just on the pip command line - pip install $(myscript.py script_with_metadata.py).
Not every little helper script needs to be a pip subcommand 😉
Very interesting discussion - I was following closely, and the last few comments gave me a pause and I think I have my own comment finally :)
Yep, I think it is not really necessary for pip to implement PEP-723 - people are definitely using pip less and less for all kind of development activities, because uv, poetry and a number of others implement all the "development-related" tooling. And in a way - that's a good thing for pip and pip maintainers.
I see the point of @pfmoore -> since pipx (that is intended to run python) and uv run (as general development-swiss-knife for Python) support it, there is generally very little strong need to add it in pip. I consider pip mostly as the "golden standard" or "reference" for the way how to install distribution packages. Full stop. Anything else is optional - and making it easier, only dillutes the main reason why we have pip.
There are other tools to run stuff, other tools to make your development environment nice and handy - and pip maintainers simply have no time to focus on those other use-cases.
I personally kept pip as "development" tool for Airflow for as long as I could (and implemented a lot of wrapping around, docs and hacks to make it work). But we decided finally (by official LAZY CONSENSUS of the Airflow community https://lists.apache.org/thread/6xxdon9lmjx3xh8zw09xc5k9jxb2n256) to switch to using uv. And that was a good choice, because uv is dedicated for that.
But we still (and likely forever will) tell our users: "This is how you install airflow: use pip" : https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html . Anything else is not directly supported and our users are free to use other tools, but we test and make sure pip installation works and this is out "golden standard" - including the way we use constraints (and we are - for example - waiting until we have a vialble PEP-751 way of installing airflow in reproducible way by pip: https://github.com/pypa/pip/issues/13334 to switch from constraints to PEP-751 lock files).
I also see that as possibly a conscious choice by pip maintainers not to implement it. If pip maintainers want to turn attention of people who want to run scripts to other tools like pipx or uv - this is a valid choice that can be made by the maintainers not to implement PEP-723 at all. This might push people more to use different tools for development or running their scripts - which might be actually a good thing. Because apparently pip main scope is not this.
Yeah, this is one reason I don't think this is worth adding to pip - it's so easy to (write a little script to) extract the dependencies from the script data, and then you can feed them into a normal
-roption or just on the pip command line -pip install $(myscript.py script_with_metadata.py).Not every little helper script needs to be a pip subcommand 😉
To be fair the same argument could have been used to not add support to install PEP 735 dependency groups, they can also be installed via pip by writing a simple Python script to output to stdout.
But now they are there they provide value as it's a single install command, and there are situations and environments where users have access to pip (e.g. using as part of a config file during a CI setup step) but running arbitrary Python is not available or requires different steps, I can see the same benefit for PEP 723 dependencies.
All that is to say I am supportive and willing to review a sufficiently high quality PR that adds PEP 723 dependencies to the pip install command (and friends), but I will not have the time to write such a PR myself.
Though, to be clear, I am also not willing to take on the reviewing all the required design choices of implementing something like pip run, and I think that would need far broader maintainer support.
When I wrote the above I didn't realize https://github.com/pypa/pip/pull/13052 was already raised, I will make some initial comments on it (it's currently not passing pre-commit or tests).
I just responded to remarks on #13052 to indicate that I'm willing to resume work on the PR. Although if anybody reading this issue wants to take a crack at the feature themselves, don't let me stop you…
That said, @pfmoore I'm hearing some possible change-of-mind between
I feel that this is a reasonable request...I wouldn’t object.
and
I don't think this is worth adding to pip - it's so easy to (write a little script to) extract the dependencies from the script data
Can you clarify if you are opposed to this feature now, or in the same "I don't really see the point but won't say no" position as in Aug 2024?
Not to put too fine a point on it, but I don't want to work on a feature that is being actively opposed by a core maintainer of pip, but I'm not sure if that's what's going on here, or if the digressions about pip run et al. have muddied the waters.
Re: "easy" and the target user's experience
The user I had in mind when I filed this feature request (and still do have in mind) is someone who is not savvy on the ephemera of Python packaging and doesn't need to be savvy, they just need a tool to work (think tired sysadmin):
- They have a
thing.py - They can see the value of having just
thing.pyinstead ofthing.py, requirements.txt... - ...as long as the steps from just-
thing.pyto "thethingis working" are simple- in particular, one command to replace the old
-r requirements.txtspelling is ideal
- in particular, one command to replace the old
In my experience, this kind of user is going to roll their eyes when they're told to install pipx or uv or [insert third-party tool] no matter how good the arguments for those things are. pip is what (probably) came with their Python and their question is going to be "well dammit, why doesn't pip support that?!"
For this user, I think that if pip doesn't support PEP 723, then PEP 723 is overall going to make their experience of Python worse and not better, because it's going to drag more tools into their lives, and they're already drowning in "easy" tools.
Can you clarify if you are opposed to this feature now, or in the same "I don't really see the point but won't say no" position as in Aug 2024?
Thanks for asking. I still think that it's so easy to write a script to extract the data in requirements format that it's not worth adding to pip. But I know that there are a lot of people who seem unwilling to write "helper utilities" like this, and for them, I'm willing to accept that (as long as someone does the work to design and add the feature) there's no real harm in adding it. That "no real harm" is difficult to quantify, of course - the biggest "harm" is that we're adding yet another obscure option to pip, and getting the UI right isn't easy (the naming of the option has already come up, for example - I don't think any of the proposed names are ideal, to be honest).
So I'm still of the view that I don't see the point, but I won't block this. I will express my concerns, but if you disagree and still feel the feature is worth it, and can get another pip maintainer to agree to merge the change, then that's OK.
The user I had in mind when I filed this feature request...
I still find this use case hard to understand. I've been in the "tired sysadmin" situation, and my experience is similar, but I don't see the value of a pip command like you describe. If you don't mind, can we unpack it a bit further?
- They have a
thing.py.
Do they just have thing.py, or do they have some other files like a requirements.txt or some instructions on how to run thing.py? Did they write thing.py themselves, or did they get it from someone else? In my experience, no-one "just" has a thing.py unless they are trying to get some hack that no-one supports running - and in that case, the sysadmin has far greater problems than working out what dependencies they need. It's easier for a separate requirements.txt file to get lost than metadata embedded in the script itself, but again, if you're installing a utility in an environment you care about, and you haven't got all the information you need, you should be asking some pointed questions of the person who supplied the utility, and not just typing pip commands and hoping (much as you might want to do that, just to get the job done).
It's also worth noting that in my experience, script dependency metadata tends to only be a few (maybe up to 5) dependencies. It's not hard to copy a list that small by hand, at a pinch. Anything with more dependencies than that is typically complex enough that it needs to be packaged more formally. Your experience may vary on this, though, and if so I'd like to hear more about why you have single-file scripts with significant dependency lists.
- They can see the value of having just
thing.pyinstead ofthing.py,requirements.txt...
I can see this. But it implies that they currently have more than just thing.py - they have a requirements file and a bunch of instructions on how to set up a virtual environment, how to install the dependencies from requirements.txt, etc. And they have (or need to have) processes to maintain that virtual environment, or create it on demand. That's a lot of work, and I can completely appreciate the sentiment "why can't I just run this script without all of that overhead?" That's precisely the use case that script metadata was designed to address.
However...
- ...as long as the steps from just-
thing.pyto "the thing is working" are simple.
This is the problem - if you're not willing to use a script runner, then the steps aren't simple. All of the environment management tasks still exist. You still need to ensure that you keep a copy of the correct "base" Python version installed[^1]. The only saving is that you can replace -r requirements.txt with --some-option thing.py when building the environment (and you still need to manually check the requires-python metadata when building the virtualenv). For me, that doesn't meet the bar of "making it simple". You say "in particular, one command to replace the old -r requirements.txt spelling is ideal" - from my experience, I disagree completely - the -r step is the least of the problems, and just replacing that is far from ideal.
[^1]: Something that's happened to me a lot - I upgrade my system Python, and all my old virtual environments stop working because the base Python has gone 🙁
Compare this with needing to install a script runner - uv, pipx, pip-run, or whatever (uv is the most full-featured, as it can manage downloading the correct Python interpreter for you as well). You have to install one tool, once. Then, all you do is uv run thing.py. Frankly, things can't get much simpler than that.
I'd completely agree that it would be better if running scripts with dependency metadata was built into core Python. But that's a debate to have with the core devs, not here - and the core devs have historically been pretty unwilling to add packaging features to the core or stdlib.
Thanks for asking. I still think that it's so easy to write a script to extract the data in requirements format that it's not worth adding to pip. But I know that there are a lot of people who seem unwilling to write "helper utilities" like this, and for them, I'm willing to accept that (as long as someone does the work to design and add the feature) there's no real harm in adding it
I am still of the opinion that this line of reasoning would have equally applied to PEP 735 – Dependency Groups in pyproject.toml, and they were accepted in to pip.
To my surprise I have already seen multiple "in the wild" uses of --group, this has increased my feeling that installing PEP 723 dependencies would also be used in the real world.
I also still think of the use cases of working in a system where setting up in an environment is something along the lines of being able to pass arguments to pip to install dependencies, but running arbitrary code is not a given option.
That "no real harm" is difficult to quantify, of course - the biggest "harm" is that we're adding yet another obscure option to pip, and getting the UI right isn't easy (the naming of the option has already come up, for example - I don't think any of the proposed names are ideal, to be honest).
Names are often hard, I don't think this is special to this feature. As there is no plan to add pip run I think the UX should be simpler than PEP 735 – Dependency Groups in pyproject.toml, the file should be explicitly specified the same way as requirements and constraints files are now, but via a dedicated option that knows they are PEP 723 files.
FWIW, if this feature is added to pip, I plan to use it in pip's own CI to dogfood this, replacing this pipx command with a the three steps of venv, pip install, and python: https://github.com/pypa/pip/blob/25.2/.github/workflows/update-rtd-redirects.yml#L28. This is just for dogfooding, but it will be used in at least one place.
I am not sure how to measure obscurity, but given I already see --group being used, I have to think some users will have common use for this feature.
So I'm still of the view that I don't see the point, but I won't block this. I will express my concerns, but if you disagree and still feel the feature is worth it, and can get another pip maintainer to agree to merge the change, then that's OK.
I am still willing to review, and accept if high enough quality and no maintainer blocks, though I am not going to try to convince another maintainer if they block based on not accepting that the feature should be added to pip at all.
the file should be explicitly specified the same way as requirements and constraints files are now, but via a dedicated option that knows they are PEP 723 files.
Alternatively, how about extending the syntax of how we specify requirement files? So it would be pip install -r script:thing.py?
Do they just have thing.py, or do they have some other files like a requirements.txt or some instructions on how to run thing.py? Did they write thing.py themselves, or did they get it from someone else? In my experience, no-one "just" has a thing.py unless they are trying to get some hack that no-one supports running
This is something I could have been clearer about, I am thinking specifically about the just thing.py case where someone's whipped up a handy script that does [whatever] and can be passed around as that one file. Admittedly, there's a lot of speculation about possible futures here.
Thanks @pfmoore for the direct answer to whether or not the feature is opposed, and for providing your perspective on that target user.
the biggest "harm" is that we're adding yet another obscure option to pip
Alternatively, how about extending the syntax of how we specify requirement files? So it would be pip install -r script:thing.py?
I see the wisdom of trying not to add a new option, but I'm also a bit superstitious about adding special semantics to -r. I wish there was a shorter form of --requirements-from-script 😅
I will note that my PR as-is checks requires-python as well which is not a package dependency but not unlike the idea of a requirement. That and the idea that inline metadata might get fancier in the future (presupposing that pip cares about that "fancier", which based on this conversation it likely would not!) is what nudged me towards the quite-generic --script.
I'm willing to go with whatever name maximizes maintainer happiness, I care more that the feature is implemented than I do about the name.
Alternatively, how about extending the syntax of how we specify requirement files? So it would be
pip install -r script:thing.py?
There is already a lot of difficulty determining a file path vs. a URL in that option. I wouldn't be against it if others really liked it, but I would want a significant amount of edge case tests.
This is something I could have been clearer about, I am thinking specifically about the just thing.py case where someone's whipped up a handy script that does [whatever] and can be passed around as that one file.
That's precisely the use case that PEP 723 was designed to support, but in the context of the data being used by a script runner, not by an installer. I still don't really see why the type of user you're imagining would find creating a virtual environment, pip installing the dependencies from the script, doing path/to/venv/python thing.py, and then either removing the virtual environment, or maintaining it for future runs, as being better than simply doing uv run thing.py[^1].
There is already a lot of difficulty determining a file path vs. a URL in that option.
No worries, it was just a thought.
I wish there was a shorter form of
--requirements-from-script
I don't actually think the longer form is that bad.
[^1]: As a pip maintainer, I feel weird recommending uv over pip, but this is very much something that uv considers in scope whereas pip doesn't.
For me, it comes down to this:
- pip is the standard way of installing python dependencies
- PEP 723 is a standard way of listing python dependencies
If I were looking at python from the outside, it would astonish me that those dots were not joined up. Of course, other things would astonish me too, but this one is pretty tractable to knock on the head.
- PEP 723 is a standard way of listing python dependencies
Just to be precise, PEP 723 is a standard way of describing what are the expectations for runnable script. This is indeed more case for pipx than for pip - because pipx is the standard way to install and run executable scripts. It's not that astonishing if you consider that pip does not even have ambitions of doing the running part.
But in general I agree, yes, having support in pip for installing dependencies from script with --requirements-from-scripts seems to be a very good idea, because there are workflows that people might want to use it with pip.
I don't actually think the longer form is that bad.
Quite agree. Explicit is better than implicit.
As a pip maintainer, I feel weird recommending uv over pip, but this is very much something that uv considers in scope whereas pip doesn't.
You could also recommend pipx run script.py without stepping out of pypa realm :)
the file should be explicitly specified the same way as requirements and constraints files are now, but via a dedicated option that knows they are PEP 723 files.
Alternatively, how about extending the syntax of how we specify requirement files? So it would be
pip install -r script:thing.py?
In terms of alternative designs that don't require a new option, what about simply pip install -r thing.py? Auto-detecting whether the file contains inline script metadata using the the reference implementation regex and if not detected falling back to a regular requirements file?
If running the regex each time is considered costly (though I doubt this compared to the already existing requirement parser), it could be limited to files that end in ".py", ".pyw", or have no "." in them on the assumption that no file extension means they are executable script command (though there would need to be some choices to make about URLs).
I'm not particularly advocating for it, it feels a little too magical to me, but if it's preferred over adding yet another option then I'd be supportive.
Auto-detecting whether the file contains inline script metadata using the the reference implementation regex and if not detected falling back to a regular requirements file? ... it feels a little too magical to me
It's easy enough to implement but I agree that it feels too magical. I plan to go forward with --requirements-from-script for now since it's been encouraged (thanks!), unless the discussion produces some stronger opinions.