pipenv
pipenv copied to clipboard
pipenv (patched pip) corrupts path for editable requirement on Windows
Issue description
I try to install a package into pipenv's virtual environment as editable (either using relative path pipenv install -e .
or absolute path pipenv install -e D:\long\windows\path\project
). I get the error with a traceback with this exception: pipenv.patched.pip._internal.exceptions.InstallationError: D:longwindowspathproject is not a valid editable requirement. It should either be a path to a local project or a VCS URL...
. (Full traceback provided below.)
With some adhoc debugging I found that the path is first resolved if it's relative, and then the absolute path gets corrupted (maybe not only) here (link):
options = shlex.split(options_str)
While running install -e ...
commands, this line executes twice:
-
options_str == '-i https://pypi.org/simple C:\\Users\\user\\AppData\\Local\\pipenv\\pipenv\\Cache'
,options == ['-i', 'https://pypi.org/simple', 'C:UsersuserAppDataLocalpipenvpipenvCache']
-
options_str == '-e D:\\long\\windows\\path\\project'
options == ['-e', 'D:longwindowspathproject']
shlex.split
for some reason removes every backslash from a Windows path, I have tested it in IDLE. I have also tried to run pipenv install -e D:/long/windows/path/project
, no changes happened.
When I tried to replicate it with just virtualenv and pip, this line in pip was never executed. (theories ahead) I suppose this is because pipenv creates a temporary requirements.txt
to supply it to pip, and somehow it goes wrong. Maybe for local files it should use URLs with file
scheme instead of plain absolute paths? ( hinted link)
Steps to replicate
-
be on Windows (10)
-
have a Python project which already has associated virtual environment managed by pipenv (I created it with
python -m pipenv --python=<python-path>
at project root)
3.1. from the root directory of the project, run python -m pipenv install -e .
(as in docs)
OR
3.2. run python -m pipenv install -e <project-absolute-path>
Expected result
The package is installed into associated virtual environment as editable.
Actual result
The package is not installed, pipenv exits with an error, producing this:
long traceback
d:\long\windows\path\project>python -m pipenv install -e . --dev --verbose
Loading .env environment variables...
Installing -e ....
Resolving -e ....
Installation Succeeded
Pipfile.lock (c606b7) out of date, updating to (916e2e)...
Locking [packages] dependencies...
Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
Traceback (most recent call last):
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 675, in <module>
main()
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 661, in main
_main(
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 645, in _main
resolve_packages(
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 612, in resolve_packages
results, resolver = resolve(
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 592, in resolve
return resolve_deps(
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 918, in resolve_deps
results, hashes, internal_resolver = actually_resolve_deps(
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 691, in actually_resolve_deps
resolver.resolve()
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 443, in resolve
constraints = self.constraints
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 401, in constraints
possible_constraints_list = self.possible_constraints
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 388, in possible_constraints
possible_constraints_list = [
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 389, in <listcomp>
install_req_from_parsed_requirement(
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 485, in install_req_from_parsed_requirement
req = install_req_from_editable(
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 240, in install_req_from_editable
parts = parse_req_from_editable(editable_req)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 209, in parse_req_from_editable
name, url, extras_override = parse_editable(editable_req)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 126, in parse_editable
raise InstallationError(
pipenv.patched.pip._internal.exceptions.InstallationError: D:longwindowspathproject is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with bzr+http, bzr+https, bzr+ssh, bzr+sftp, bzr+ftp, bzr+lp, bzr+file, git+http, git+https, git+ssh, git+git, git+file, hg+file, hg+http, hg+https, hg+ssh, hg+static-http, svn+ssh, svn+http, svn+https, svn+svn, svn+file).
Locking Failed!
[ ] Locking...
Traceback (most recent call last):
File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\__main__.py", line 4, in <module>
cli()
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\options.py", line 58, in main
return super().main(*args, **kwargs, windows_expand_args=False)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1078, in main
rv = self.invoke(ctx)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\decorators.py", line 92, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\command.py", line 209, in install
do_install(
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 297, in do_install
raise e
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 281, in do_install
do_init(
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 648, in do_init
do_lock(
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\lock.py", line 65, in do_lock
venv_resolve_deps(
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 859, in venv_resolve_deps
c = resolve(cmd, st, project=project)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 728, in resolve
raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!
$ pipenv --support
Pipenv version: '2023.11.17'
Pipenv location: 'C:\\Users\\user\\AppData\\Roaming\\Python\\Python310\\site-packages\\pipenv'
Python location: 'C:\\Program Files\\Python310\\python.exe'
OS Name: 'nt'
User pip version: '23.3.2'
user Python installations found:
PEP 508 Information:
{'implementation_name': 'cpython',
'implementation_version': '3.10.11',
'os_name': 'nt',
'platform_machine': 'AMD64',
'platform_python_implementation': 'CPython',
'platform_release': '10',
'platform_system': 'Windows',
'platform_version': '10.0.19045',
'python_full_version': '3.10.11',
'python_version': '3.10',
'sys_platform': 'win32'}
System environment variables:
...
-
PIP_DISABLE_PIP_VERSION_CHECK
-
PYTHONDONTWRITEBYTECODE
-
PIPENV_VENV_IN_PROJECT
-
PYTHONFINDER_IGNORE_UNSUPPORTED
Pipenv-specific environment variables:
-
PIPENV_VENV_IN_PROJECT
:1
Debug-specific environment variables:
-
PATH
:...;C:\Program Files\Python310\Scripts\;C:\Program Files\Python310\;...;C:\Windows\System32;C:\Windows;C:\Windows\System32\wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;...;C:\Users\user\AppData\Roaming\Python\Python310\Scripts;...;
Contents of Pipfile
('D:\long\windows\path\project\Pipfile'):
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
[dev-packages]
[requires]
python_version = "3.10"
python_full_version = "3.10.11"
Contents of Pipfile.lock
('D:\long\windows\path\project\Pipfile.lock'):
{
"_meta": {
"hash": {
"sha256": "..."
},
"pipfile-spec": 6,
"requires": {
"python_full_version": "3.10.11",
"python_version": "3.10"
},
"sources": [
{
"name": "pypi",
"url": "https://pypi.org/simple",
"verify_ssl": true
}
]
},
"default": {},
"develop": {}
}
Please try specifying the name@
prefix syntax before specifying your editable requirement, I think its struggling on identifying the correct name from the project config for your library.
I also tried to specify path on command as file
-scheme URL, it got resolved to the same D:longwindowspathproject
.
Is it like this...
d:\long\windows\path\project>python -m pipenv install --dev -e package@"." --verbose
output
Loading .env environment variables...
Installing -e package@....
Resolving -e package@....
Traceback (most recent call last):
File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\__main__.py", line 4, in <module>
cli()
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\options.py", line 58, in main
return super().main(*args, **kwargs, windows_expand_args=False)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1078, in main
rv = self.invoke(ctx)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\decorators.py", line 92, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\command.py", line 209, in install
do_install(
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 212, in do_install
pkg_requirement, _ = expansive_install_req_from_line(
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\dependencies.py", line 962, in expansive_install_req_from_line
return install_req_from_editable(pip_line, line_source), name
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 240, in install_req_from_editable
parts = parse_req_from_editable(editable_req)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 209, in parse_req_from_editable
name, url, extras_override = parse_editable(editable_req)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\req\constructors.py", line 126, in parse_editable
raise InstallationError(
pipenv.patched.pip._internal.exceptions.InstallationError: package@. is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with bzr+http, bzr+https, bzr+ssh, bzr+sftp, bzr+ftp, bzr+lp, bzr+file, git+http, git+https, git+ssh, git+git, git+file, hg+file, hg+http, hg+https, hg+ssh, hg+static-http, svn+ssh, svn+http, svn+https, svn+svn, svn+file).
Same with an absolute path, but the exception text is pipenv.patched.pip._internal.exceptions.InstallationError: package@D:\long\windows\path\project is not a valid editable requirement...
Hmmm, I'll have to think on this more, just wondering though which shell, is it windows power shell? I want to be able to reproduce the issue.
I used cmd.exe on Windows 10 initially.
I just tried python -m pipenv install -e .
in powershell, it got installed on first try. I could not replicate the problem in powershell, though I tried it several times both from cmd.exe (with the error) and powershell (successfully, cleaning Pipfile and lockfile afterwards).
I don't know powershell stuff, printf debugging doesn't work well... That corrupting line is executed twice:
-
options_str == '-i https://pypi.org/simple C:\\Users\\user\\AppData\\Local\\pipenv\\pipenv\\Cache'
options == ['-i', 'https://pypi.org/simple', 'C:UsersuserAppDataLocalpipenvpipenvCache']
-
options_str == '-e .'
options == ['-e', '.']
Also, even when I specify the absolute path on the same drive, it comes to this line as relative. If it's the same path and the relative is .
, installation proceeds without errors. If it's a different path, the same exception pipenv.patched.pip._internal.exceptions.InstallationError
occurs with corrupted relative path.
So the problem with shlex
persists, but since the path .
is not resolved when running from powershell (for some reason), it does not cause a visible error.
Also, when I specify the absolute path on a different drive, I get this:
command, output
PS D:\long\windows\path\project> python -m pipenv install -e "C:\Users\user\project\"
Loading .env environment variables...
Installing -e C:\Users\user\package\...
Resolving -e C:\Users\user\package\...
Traceback (most recent call last):
File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\__main__.py", line 4, in <module>
cli()
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\options.py", line 58, in main
return super().main(*args, **kwargs, windows_expand_args=False)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1078, in main
rv = self.invoke(ctx)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\decorators.py", line 92, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\command.py", line 209, in install
do_install(
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 255, in do_install
added, cat, normalized_name = project.add_package_to_pipfile(
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\project.py", line 1211, in add_package_to_pipfile
name, normalized_name, entry = self.generate_package_pipfile_entry(
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\project.py", line 1155, in generate_package_pipfile_entry
req_name = determine_package_name(package)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\dependencies.py", line 792, in determine_package_name
req_name = find_package_name_from_directory(package.link.file_path)
File "C:\Users\user\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\dependencies.py", line 644, in find_package_name_from_directory
os.listdir(directory),
FileNotFoundError: [WinError 3] The system cannot find the path specified: '\\Users\\user\\project'
It happens because here in pipenv the original absolute path 'C:\\Users\\user\\project'
is supplied to urllib.parse.urlparse
, which treats drive letter as URL scheme c
, and the following code removes this scheme. I have no idea what's going on there.
I checked the cache dir which path is also corrupted: any recent files are related to my other work with pipenv where I don't use editable dependencies.
example to reproduce:
- be on Windows (Windows 10)
- clone this repo into
pipenv-issue-6068-demo
(commit 2131b01d7ed7c723261f0e4ba298a46524dad41e ) - open cmd.exe
-
cd pipenv-issue-6068-demo
-
python -m pipenv install -e .
(I run Python 3.10, pipenv 2023.11.17)
this produces the error from the initial comment, with exception: pipenv.patched.pip._internal.exceptions.InstallationError: D:longpathpipenv-issue-6068-demo is not a valid editable requirement. It should either be a path to a local project or a VCS URL ...
@asrelo thanks for providing an example -- I am checking it out, but I do see that the local file install doesn't have a pyproject.toml, or setup.cfg or setup.py so technically I don't think its installable -- are you actually able to pip install that as a local package?
There is pyproject.toml
at root of pipenv-issue-6068-demo, I'm not aware what is wrong with that.
Yes, virtualenv
+ pip install -e .
successfully installs the package. This was the session from within this same repo in cmd.exe:
cmd.exe session
d:\long\windows\path\pipenv-issue-6068-demo>python -m virtualenv testenv
created virtual environment CPython3.10.11.final.0-64 in 7748ms
creator CPython3Windows(dest=D:\long\windows\path\pipenv-issue-6068-demo\testenv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\user\AppData\Local\pypa\virtualenv)
added seed packages: pip==23.3.2, setuptools==69.0.3, wheel==0.42.0
activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
d:\long\windows\path\pipenv-issue-6068-demo>.\testenv\Scripts\activate
(testenv) d:\long\windows\path\pipenv-issue-6068-demo>pip install -e .
Obtaining file:///D:/long/windows/path/pipenv-issue-6068-demo
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Installing backend dependencies ... done
Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: problem-project
Building editable for problem-project (pyproject.toml) ... done
Created wheel for problem-project: filename=problem_project-0.1.0-0.editable-py3-none-any.whl size=1594 sha256=f76979778a63df8472b46c266f35d72fefe6ea130c3bf8329b955e1a92c2a68f
Stored in directory: C:\Users\user\AppData\Local\Temp\pip-ephem-wheel-cache-q_w0s3yo\wheels\ab\55\bc\7e8acf252f651d596add8fe1a7bd9aa460d7f1424001943dd5
Successfully built problem-project
Installing collected packages: problem-project
Successfully installed problem-project-0.1.0
(testenv) d:\long\windows\path\pipenv-issue-6068-demo>problem_package
Hello World!
upd: and it is actually editable
git bash, cmd.exe and powershell are all working on mine:
C:\Users\matte\Projects\pipenv-triage\pipenv-issue-6068-demo>pipenv install .
Loading .env environment variables...
Installing ....
Resolving ....
Installation Succeeded
Installing dependencies from Pipfile.lock (95c512)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
C:\Users\matte\Projects\pipenv-triage\pipenv-issue-6068-demo>pipenv run problem_package
Loading .env environment variables...
Hello World!
🤔 Only change I made to your example was to allow python 3.11 so I wouldn't have to edit my PATH for the python3.10 version.
I just tried with freshly installed Python 3.11.7 and pipenv 2023.11.17. The error was the same, only the traceback was fancier...
I'll try to reproduce the example on a different machine soon (though shlex
itself worked there exactly the same).
Tried to reproduce with another person. The initial error with pipenv install -e .
does not happen; shlex.split
there receives relative path .
and proceeds without errors. I have no clue why that path .
gets resolved to absolute path only when pipenv is running from my cmd.exe.
However, pipenv install -e <relative-path>
does not work, as in (comment):
If it's a different path, the same
exception pipenv.patched.pip._internal.exceptions.InstallationError
occurs with corrupted relative path.
Cache path gets corrupted too.
We didn't try to specify a path on a different drive as in the comment, now I humbly assume there would be the same error as before.
@asrelo I believe that is because the relative path is now the src directory which is missing the pyproject.toml (its in the parent).
I mean, I have the fresh project dir at D:\long\windows\path\project
with no pipenv's virtualenv and no pipfiles. I copy the whole project
dir to D:\long\project
, either renaming project and package or not.
From within original project path, I run python -m pipenv install -e ..\..\..\project
(or python -m pipenv install -e D:\long\project
, but it comes to shlex.split
as relative path still). It fails because pipenv tries to find something at ......project
(comment where this particular error is first mentioned).
And absolute paths on different drives don't work as well.
I expect that it should work because pipenv --help
output and man page do not mention particular limitations on a local path for the -e
option, and because it works with pip itself as I just tested.