pip-tools
pip-tools copied to clipboard
Using `pip-tools` as an API?
Hi there! I'm reaching out as the lead engineer working on pip-audit, a tool that Google and Trail of Bits are developing to allow developers to audit their Python environments for packages with known vulnerabilities
One of our goals is to support a requirements-file mode, and pip-tools
' support for resolving entire dependency trees and pinning versions seemed like a strong choice for that task.
What's the problem this feature will solve?
Currently, pip-tools
exposes two CLI tools: pip-compile
and pip-sync
. These work great, but they're difficult to interact with directly via Python: in order to utilize pip-compile
's dependency resolution, we'd need to create a subprocess, control where pip-compile
puts its outputs, and then read those outputs back in.
It would be great if we could skip that subprocess step and import pip-tools
directly as a Python module!
Describe the solution you'd like
Our ideal solution looks something like this rough sketch:
import pip_tools
resolved_dep_list = pip_tools.compile(some_requirements_string)
(Where some_requirements_string
is a string in Requirements File Format and resolved_dep_list
is something resembling a List[Tuple[str, Version]]
.
Alternative Solutions
As mentioned above: the next best thing for our use case is probably to use a subshell and hack together an environment wherein we're relatively confident that running pip-compile <some-input>
won't cause any externally observable side effects. But we think that a real Python API would be a lot cleaner, and a net win for the packaging ecosystem!
Additional context
The work we're doing on pip-audit
is currently funded, and we have budget set aside for making changes/improvements to the Python packaging tooling ecosystem. We're happy to use some of our engineering time here, if it would help!
I think this ought to be considered related to #1377 (dependent on?), and I encourage you to help build a consensus on the ideal structured output object there.
In the hacks department, here are two examples with plumbum:
In [1]: from plumbum.cmd import pip_compile
In [2]: deps = (pip_compile['-', '-o', '-'] << "requests")()
In [3]: print(deps)
#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
# pip-compile --output-file=- -
#
certifi==2021.5.30
# via requests
charset-normalizer==2.0.6
# via requests
idna==3.2
# via requests
requests==2.26.0
# via -r -
urllib3==1.26.6
# via requests
In [1]: from plumbum import local
In [2]: pip_compile = lambda reqsin: [tuple(line.split('==')) for line in (local['pip-compile']['--no-annotate', '--no-header', '-', '-o', '-'] << reqsin)().splitlines()]
In [3]: deps = pip_compile("requests")
In [4]: print(deps)
[('certifi', '2021.5.30'), ('charset-normalizer', '2.0.6'), ('idna', '3.2'), ('requests', '2.26.0'), ('urllib3', '1.26.6')]
Thanks for following up! I'll go ahead and opine on #1377 as well, but FWIW: our ideal output for an API would be nearly identical to the one currently produced by the CLI; something to the effect of List[Requirement]
, where Requirement
is or is similar to packaging.requirements.Requirement
from the PyPA's packaging
package.
our ideal output for an API would be nearly identical to the one currently produced by the CLI
What if this gets replaced by the lockfile proposal currently being discussed on dpo, that would fulfill the request, right?
What if this gets replaced by the lockfile proposal currently being discussed on dpo, that would fulfill the request, right?
Yes! A full standard lockfile format would fulfill this and make changes to pip-tools
completely unnecessary 🙂