pip-tools
pip-tools copied to clipboard
jazzband
pip-compile absolute links when given relative path
requirements.in:
-e .
-e ./sqlalchemy_jsonapi
compiles to:
-e file:///Users/coltonprovias/MyProject
-e file:///Users/coltonprovias/MyProject/sqlalchemy_jsonapi
It should compile to
-e .
-e ./sqlalchemy_jsonapi
It's not really useful for me to provide a requirements.txt file then to anybody unless they use a Mac and have the same name as me. Although if that person does exist, I would like to meet them.
This has affected me too as I'm trying to install packages which are submodules of my project. It seems the issues arises here: https://github.com/pypa/pip/blob/develop/pip/req/req_install.py#L1131
I suspect it is hard to avoid as pip-tools uses the pip library and the pip library has no reason to keep a relative path around. It wants to deal with absolute paths.
Which means that it is unlikely that we'd get the pip library changed to help us so which suggests that we'd have to get the output from the pip library and go back and make any paths relative again which is an option but not incredibly desirable as it might be hard to know exactly when to do it and when not to.
Personally, I'm going to try switching from using submodules and relative paths to having git+git:// paths in my requirements.in file but at the same time it seems very reasonable to try to fix this as it makes little sense to have absolute paths in a requirements.txt file when there is little chance that the paths in different development environments or the production environment is going to be the same. That is certainly what affects me. Of course, the requirements.txt should also be in git so it should be possible to pip-compile and then edit the requirements.txt before committing it.
I've been planning to move all of the requirements handling across all of our repos to using pip-tools and just realized it had this issue, which may completely shelve that plan. This is a pretty significant problem.
So the core of this comes down to this function here, inside pip. I don't think it would be easy to fix in pip itself, so I will see if I can manipulate the result while outputting it to the file by making the result relative.
the relative paths should actually use file:foo/bar so that the #egg=spam fragment could be appended. Unfortunately this breaks in pip-compile, as pip-compile tries reading foo/bar/setup.py after setting foo/bar as the current directory!
as for why #egg=spam would be needed, is for pip dependency resolver to find out what abstract requirement is provided by the said file link! Otherwise it is going to attempt to download spam from Warehouse!
If your parrot is a git submodule, this works well: -e git+path/to/dead/parrot#egg=parrot.
@rollcat almost works... except that we merged 2 repos so it is a slug and hardly speaks.
I took another stab at trying to fix this in https://github.com/jazzband/pip-tools/pull/702.
I've been using the following post-processing step to get around the problem: make_paths_relative.py < requirements.txt (script below)
#!/usr/bin/env python
import os
import re
import sys
pattern = re.compile(r"\s*-e\s+file://(?P<path>.*?)\n")
inputs = sys.stdin.readlines()
for line in inputs:
match = pattern.match(line)
if match:
path = match.group("path")
path = os.path.relpath(path)
line = f"-e {path}\n"
sys.stdout.write(line)
A(nother) hacky solution, in requirements.in:
-e file:relative/path/to/my_package
then to compile:
pip-compile -q -o - | sed s:///::g > requirements.txt
Ran into this issue as well. If I modify all my -e src/foo links to -e file:src/foo#egg=foo then the compiled requirements file drops out the same -e file:src/foo#egg=foo and this works as expected. Quite a lot more verbose than necessary though. The egg on the end is required, or the sed script above in https://github.com/jazzband/pip-tools/issues/204#issuecomment-486242541 otherwise the later pip install -r will not properly find the "foo" project to satisfy the dependency lists of other projects depending on "foo".
@mmerickel
Ran into this issue as well. If I modify all my
-e src/foolinks to-e file:src/foo#egg=foothen the compiled requirements file drops out the same-e file:src/foo#egg=fooand this works as expected.
That's a nice workaround!
I don't see it mentioned specifically here, but I've had the same issue with --find-links ./local_src. I want pip to look for packages within a directory as part of the project, but pip-compile translates that into an absolute path with './local_src' tacked on to the end.
@tisdall you could try --find-links file:./local_src. It keeps the relative path.
@atugushev thanks! Strangely when I tried it I had both relative and the original full path until I completely deleted the 'requirements.txt' and then recompiled.
I've pinned the issue to bring more attention to the most wanted feature based on reactions.
I'm happy to work on this and other path-sanity tasks, but would like some discussion re: #1067 before diving in.
Just ran into this issue today when migrating to pip-tools while still listing requirements inside setup.py, blocking our implementation.
I was trying to figure out the proper syntaxes supported for paths, by reading here.
That indicates either of the following are acceptable:
[-e] [<vcs>+]file://<path>[#egg=<name>]
[-e] <name> @ file://<path>
Notes on that:
- Be careful, if you accidentally add a slash to a relative path (
file:///.) you'll be trying to install a whole lot more than you bargained for! - It seems relative paths here aren't supported by
file://; if I havefile://.#egg=pip-toolsand trypip install -r requirements.in, it'll fail with:
ValueError: non-local file URIs are not supported on this platform: 'file://.'
- Using the undocumented (at least at the above link) syntax
file:.will work with the first syntax, but not the second. - Using
git+file:.will not work. -
git+file://will not work with the second syntax.
So my understanding is now that these are the appropriate syntaxes for local paths:
[-e] [<vcs>+]file://<absolute>[#egg=<name>]
[-e] <name> @ file://<absolute>
[-e] file:<relative>[#egg=<name>]
Q1: Is this an acceptable standard for this project?
I haven't begun to look at the setup.py syntaxes.
Q2: Setting aside setup.py for the moment, if we abide by the above potential standard, what doesn't work?
Q3: I also want to note and check for agreement here that we should ensure the relative paths output are reformulated to be relative to the output file, if output "file" is not stdout.
Hello @zuckerruebe,
Thanks for pinging! As far as I know, nobody works on this issue at the moment. In the meantime, you might use these workarounds:
- https://github.com/jazzband/pip-tools/issues/204#issuecomment-564597799
- https://github.com/jazzband/pip-tools/issues/204#issuecomment-550051424
@zuckerruebe @atugushev and anyone else, can you provide any feedback for Q1, Q2, Q3 at the end of my last comment (jump to "So my understanding" above the last code block)?
Q1: Is this an acceptable standard for this project?
IMO Instead
[-e] file:<relative>[#egg=<name>]
it should be:
[-e] <relative>[#egg=<name>]
AFAIK file:./relative/path is not a standard (more like a workaround) and we shouldn't rely on it in a long prospect future. I swear I saw a discussion about it on pip tracker, but can't find anymore :(
Q2: Setting aside
setup.pyfor the moment, if we abide by the above potential standard, what doesn't work?
I think I'm not following the question here. Could you elaborate?
Q3: I also want to note and check for agreement here that we should ensure the relative paths output are reformulated to be relative to the output file, if output "file" is not stdout.
Yes, makes sense to me.
UPDATE: found a discussion about file:./relative/path https://github.com/pypa/packaging/issues/120
Thanks @atugushev!
Q2
I'm asking: if users use one of the following syntaxes:
[-e] [<vcs>+]file://<absolute>[#egg=<name>]
[-e] <name> @ file://<absolute>
[-e] file:<relative>[#egg=<name>]
what exactly goes wrong, as the most basic case of the issue (compiling relpaths to relpaths) doesn't seem to be a problem? I notice two things to start with:
- only compiles if done from the same directory as the in-file
- the relpath is preserved literally, anchored to the path of the in-file, instead of the out-file
Q1
After reading the pypa/packaging issue you linked (and spidering out from there), it looks like we're waiting for some things to shake out at pypa/pip and pypa/packaging:
- this issue will not be thoroughly resolved until at least:
- pypa/pip#6658
- pypa/packaging#120
- we should also be interested to track:
- pypa/pip#7650
- pypa/packaging#264
- PEP 610
- pypa/pip#5204
- pip's test_basic_install_relative_directory
- you are right,
file:./relpathis an unreliable non-standard, not to be encouraged (but it is tested for in pip'stest_basic_install_relative_directory) - but
[-e] <relative>[#egg=<name>]is not great either. From pip#6658 (comment):[simply
./some/path] wouldn't be acceptable per PEP 508, so it would be hard to justify supporting much less get it working across all tools. - I think we're waiting on better support for
name @ <relpath>, and generally stronger consensus on proper spec -- relative paths are currently resolved based on different origins depending on tool/codepath...
We may eventually be able to focus on:
[-e] <name> @ [git+file://]<relative-or-absolute>[@<tag-ish>]
I just want to reiterate for all the watchers that while we're still waiting on path syntaxes to settle upstream, much of the time, for now, you'll get what you want by using file:./relpath, such as -e file:..
I welcome any and all testers to try #1329, and please let me know if you consider this issue fixed by its changes.