dulwich
dulwich copied to clipboard
Improve symlinks on Windows
When trying to clone the library pyglossary from git using dulwich, I get the following error:
Enumerating objects: 18119, done.
Counting objects: 100% (102/102), done.
Compressing objects: 100% (78/78), done.
Total 18119 (delta 53), reused 57 (delta 23), pack-reused 18017
Traceback (most recent call last):
File "C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-YNNX2TaR-py3.10\lib\site-packages\dulwich\client.py", line 769, in clone
target.reset_index()
File "C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-YNNX2TaR-py3.10\lib\site-packages\dulwich\repo.py", line 1526, in reset_index
return build_index_from_tree(
File "C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-YNNX2TaR-py3.10\lib\site-packages\dulwich\index.py", line 689, in build_index_from_tree
st = build_file_from_blob(
File "C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-YNNX2TaR-py3.10\lib\site-packages\dulwich\index.py", line 602, in build_file_from_blob
os.symlink(contents, target_path)
OSError: [WinError 1314] Клиент не обладает требуемыми правами: '../../check-style' -> 'pyglossary-clone\\pyglossary\\plugins\\check-style'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\Users\hanne\Documents\Programme\dulwich-test\dulwich_test\dtest.py", line 3, in <module>
porcelain.clone("https://github.com/ilius/pyglossary.git", "pyglossary-clone")
File "C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-YNNX2TaR-py3.10\lib\site-packages\dulwich\porcelain.py", line 445, in clone
return client.clone(
File "C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-YNNX2TaR-py3.10\lib\site-packages\dulwich\client.py", line 775, in clone
shutil.rmtree(target_path)
File "C:\Python310\lib\shutil.py", line 739, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Python310\lib\shutil.py", line 612, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Python310\lib\shutil.py", line 612, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Python310\lib\shutil.py", line 612, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Python310\lib\shutil.py", line 617, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Python310\lib\shutil.py", line 615, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 32] Процесс не может получить доступ к файлу, так как этот файл занят другим процессом: 'pyglossary-clone\\.git\\objects\\pack\\tmp_pack_EMkDJk'
(Originally I reported this on Poetry): This is because the project folder contains a symbolic link and the ususual git installation for Windows does not try to create them by default, unlike dulwich. To quote my post:
The reason seems to be that by default git for windows does not create real symlinks, but instead some sort of text files. Windows apparently does not like non-admin users having the right to create symlinks because they enable symlink attacks. I don't know how much of a problem they are, maybe MS also don't like potential bugs because symlinks are less stable than on Linux.
The problem can apparently be fixed by enabling Windows developer mode, which allows symlinks even for non-admin users
I think that most developers on Windows don't know that this developer mode even exists and don't have it enabled and executing poetry with admin rights does not seem like a good solution (correct me if I'm wrong).
I think it would be great to have a better error message here that says that you don't have permissions to create symlinks and that you can enable Windows developer mode to make it work. Currently it is not that easy to figure out what is wrong.
Thanks for the report; I'm open to a PR improving the error message.
I tried adding a small message on my fork here: https://github.com/Vuizur/dulwich
Although unfortunately I tried testing it and couldn't build wheels on windows for some reason when installing from Git, so I don't know if it works. (I also tried installing the build tools that are linked in the error.)
C:\Users\hanne\Documents\Programme\dulwich-test-2> poetry add git+https://github.com/Vuizur/dulwich
Updating dependencies
Resolving dependencies...
Writing lock file
Package operations: 1 install, 0 updates, 0 removals
• Installing dulwich (0.20.45 1b4080d)
CalledProcessError
Command 'C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-2-YdXoDw1F-py3.10\Scripts\python.exe C:\Users\hanne\AppData\Roaming\pypoetry\venv\lib\site-packages\virtualenv\seed\wheels\embed\pip-22.2.2-py3-none-any.whl\pip install --use-pep517 --disable-pip-version-check --prefix C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-2-YdXoDw1F-py3.10 --upgrade --no-deps C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-2-YdXoDw1F-py3.10\src\dulwich' returned non-zero exit status 1.
at C:\Python310\lib\subprocess.py:524 in run
520│ # We don't call process.wait() as .__exit__ does that for us.
521│ raise
522│ retcode = process.poll()
523│ if check and retcode:
→ 524│ raise CalledProcessError(retcode, process.args,
525│ output=stdout, stderr=stderr)
526│ return CompletedProcess(process.args, retcode, stdout, stderr)
527│
528│
The following error occurred when trying to handle this error:
EnvCommandError
Command C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-2-YdXoDw1F-py3.10\Scripts\python.exe C:\Users\hanne\AppData\Roaming\pypoetry\venv\lib\site-packages\virtualenv\seed\wheels\embed\pip-22.2.2-py3-none-any.whl\pip install --use-pep517 --disable-pip-version-check --prefix C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-2-YdXoDw1F-py3.10 --upgrade --no-deps C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-2-YdXoDw1F-py3.10\src\dulwich
errored with the following return code 1, and output:
Processing c:\users\hanne\appdata\local\pypoetry\cache\virtualenvs\dulwich-test-2-ydxodw1f-py3.10\src\dulwich
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: dulwich
Building wheel for dulwich (pyproject.toml): started
Building wheel for dulwich (pyproject.toml): finished with status 'error'
error: subprocess-exited-with-error
Building wheel for dulwich (pyproject.toml) did not run successfully.
exit code: 1
[19 lines of output]
running bdist_wheel
running build
running build_py
copying dulwich\index.py -> build\lib.win-amd64-cpython-310\dulwich
running egg_info
writing dulwich.egg-info\PKG-INFO
writing dependency_links to dulwich.egg-info\dependency_links.txt
writing entry points to dulwich.egg-info\entry_points.txt
writing requirements to dulwich.egg-info\requires.txt
writing top-level names to dulwich.egg-info\top_level.txt
reading manifest file 'dulwich.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no directories found matching 'dulwich\tests\data'
adding license file 'COPYING'
adding license file 'AUTHORS'
writing manifest file 'dulwich.egg-info\SOURCES.txt'
running build_ext
building 'dulwich._objects' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft
C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
[end of output]
note: This error originates from a subprocess, and is likely not a problem with
pip.
ERROR: Failed building wheel for dulwich
Failed to build dulwich
ERROR: Could not build wheels for dulwich, which is required to install pyproject.toml-based projects
at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\env.py:1473 in _run
1469│ output = subprocess.check_output(
1470│ command, stderr=subprocess.STDOUT, env=env, **kwargs 1471│ )
1472│ except CalledProcessError as e:
→ 1473│ raise EnvCommandError(e, input=input_)
1474│
1475│ return decode(output)
1476│
1477│ def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
The following error occurred when trying to handle this error:
PoetryException
Failed to install C:/Users/hanne/AppData/Local/pypoetry/Cache/virtualenvs/dulwich-test-2-YdXoDw1F-py3.10/src/dulwich
at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\pip.py:51 in pip_install
47│
48│ try:
49│ return environment.run_pip(*args)
50│ except EnvCommandError as e:
→ 51│ raise PoetryException(f"Failed to install {path.as_posix()}") from e
52│
I tried adding a small message on my fork here: https://github.com/Vuizur/dulwich
Although unfortunately I tried testing it and couldn't build wheels on windows for some reason when installing from Git, so I don't know if it works. (I also tried installing the build tools that are linked in the error.)
This appears to be happening because you don't have a C environment installed for building the C extensions.
I've left some comments on the branch.
I am having the same problem here when using Poetry, which at the time uses [email protected].
I'm still open to PRs that improve the error message. Alternatively, we could allow passing in another function to be called for creating symlinks - so you could e.g. create regular files instead.