pip-tools
pip-tools copied to clipboard
Allow excluding certain packages from pip-sync
What's the problem this feature will solve?
When running pip-sync
, it will uninstall also editable package(s) and you need to pip install -e .
them again. This adds unnecessary steps. Adding support for pip-sync --ignore <packages...>
or similar would allow to avoid that.
This topic has been discussed in https://github.com/jazzband/pip-tools/issues/686. However, that one was closed with the assumption that if non-declared packages don't need to be touched, vanilla pip install -r requirements.txt
will do.
The difference here is that while I do want to uninstall all the non-declared packages, I also want to retain the editable packages.
Describe the solution you'd like
Maybe:
pip-sync --ignore "(my-package1|my-package2)" # regex
pip-sync --ignore my-package1 --ignore my-package2
My problem is specifically with editable packages. So it also could be:
pip-sync --ignore-editable
...but I'm not sure how difficult it would be find out which of the packages are actually editable and which are not. Also, former suggestion is more generic, should someone want to exclude actual, non-editable, packages.
Alternative Solutions
My workaround currently is to simply run pip install -e .
always after pip-sync
.
Even with that workaround, thouhg, pip-sync
effectively breaks if you want to run it a second time.
In other words, this workflow doesn't actually work:
$ # clone repo
$ pip-compile requirements.in
$ pip-sync requirements.txt
$ pip install -e .
# do some development
# add a new dependency to requirements.in
$ pip-compile requirements.in
$ pip-sync requirements.txt
...
AttributeError: 'NoneType' object has no attribute 'specifier'
So there's a hole to be filled here...
Missing from this discussion right now is the option of putting your editable requirements in requirements.in
with all the others.
Disadvantages:
- The absolute path will end up in
requirements.txt
if using e.g.-e .
(#204) [Fixed by unmerged #1329]- You can use
-e file:.
or-e file:.#egg=pip-tools
, and the path will stay relative through compilation, but this is not a reliable standard (pypa/packaging#120)
- You can use
- You must compile from the same directory as the in-file (not usually a problem) [Fixed by unmerged #1329]
- The relpath is relative to the in-file, not the out-file (not usually a problem) [Fixed by unmerged #1329]
- Technically it will get removed and reinstalled on each sync, but that should be near-instant since it's mostly linking around (right?)
- In the big picture, "editable"/
-e
package installation is not yet well defined for the newpyproject.toml
era, AFAIK. For example,flit
projects don't support-e
installation. When it's the project you're working on, you can useflit install -s
, but I don't know about installing dependencies this way in a pip-tools friendly way.
See my last comment from #204 for a more thorough summary of the relpath problems and links to relevant issues further upstream.
All that said, for now, it will probably "just work" if you put your editable reqs in the in-file, with a form like
-e file:RELATIVEPATH#egg=PACKAGENAME
or just
-e file:RELATIVEPATH
@AndydeCleyre I thought I had tried that, but I was doing -e .
, not -e file:.
. That makes all the difference! Thanks!
@tomasaschan If I can get #1329 merged in, the -e .
syntax should work as well.
@tuukkamustonen Do you think putting your editable requirements in your in-file adequately serves your purpose?
@AndydeCleyre I have yet to try it, but if it works I think it's a feasible workaround.
I mean, better would be to allow ignoring certain packages as suggested, because people's workflows differ. Someone might not want to initialize all sub-projects always, and avoiding that would require splitting editable packages to several different files. Or maybe people don't use pip install -e .
but python setup.py develop
(if that's still supported) or some other way.
But yes indeed, your proposal (-e file:...
) looks viable 👍🏻.
Oops, accidentally clicked the wrong button.
Was about to comment that I finally tried -e file:
and indeed it works nicely.
However, like already mentioned there are few drawback in packages getting uninstalled/re-installed with pip-sync
. Something else is that pip-sync
now always does that, even if I want to manage my editable packages via other tooling/commands. pip-sync --ignore-editable
would nicely tackle that.
I'll be more keen on --ignore-editable
or similar if/when upstream standardizes what it is to be "editable."
Meanwhile, let's iron out the essentials.
- Does anyone here want to be able to ignore some editable packages and not others, in the same project?
- Does anyone here want to ignore non-editable packages?
Another workaround, kind of a "hook":
psync () { # [<pip-sync-arg>...]
pip-sync $@
if [[ -r post_pip_sync.sh ]]; then
./post_pip_sync.sh
fi
}
And you put your custom commands for installing editables and whatever in that script.
I would prefer pip-sync
to ignore editable installs by default and offers a --include-editable
rather than a --ignore-editable
flag.
This way, if I add -e file:.
to requirements.in
, I would use the --include-editable
flag only once when I set up my environment, but afterwards I can use pip-sync
normally without having to add --ignore-editable
each time.
Alternatively, I would also be fine if there is no extra flag and pip-sync
ignores editable installs. I don't mind typing pip install -e .
once to set up the environment.
BTW, does PEP660 help with this? The aforementioned flit supports it as well.