Improve symlinks on Windows

[Vuizur]

When trying to clone the library pyglossary from git using dulwich, I get the following error:

Enumerating objects: 18119, done.
Counting objects: 100% (102/102), done.
Compressing objects: 100% (78/78), done.
Total 18119 (delta 53), reused 57 (delta 23), pack-reused 18017
Traceback (most recent call last):
  File "C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-YNNX2TaR-py3.10\lib\site-packages\dulwich\client.py", line 769, in clone
    target.reset_index()
  File "C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-YNNX2TaR-py3.10\lib\site-packages\dulwich\repo.py", line 1526, in reset_index
    return build_index_from_tree(
  File "C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-YNNX2TaR-py3.10\lib\site-packages\dulwich\index.py", line 689, in build_index_from_tree
    st = build_file_from_blob(
  File "C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-YNNX2TaR-py3.10\lib\site-packages\dulwich\index.py", line 602, in build_file_from_blob
    os.symlink(contents, target_path)
OSError: [WinError 1314] Клиент не обладает требуемыми правами: '../../check-style' -> 'pyglossary-clone\\pyglossary\\plugins\\check-style'   

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\Users\hanne\Documents\Programme\dulwich-test\dulwich_test\dtest.py", line 3, in <module>
    porcelain.clone("https://github.com/ilius/pyglossary.git", "pyglossary-clone")
  File "C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-YNNX2TaR-py3.10\lib\site-packages\dulwich\porcelain.py", line 445, in clone
    return client.clone(
  File "C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-YNNX2TaR-py3.10\lib\site-packages\dulwich\client.py", line 775, in clone
    shutil.rmtree(target_path)
  File "C:\Python310\lib\shutil.py", line 739, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Python310\lib\shutil.py", line 612, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Python310\lib\shutil.py", line 612, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Python310\lib\shutil.py", line 612, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Python310\lib\shutil.py", line 617, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Python310\lib\shutil.py", line 615, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] Процесс не может получить доступ к файлу, так как этот файл занят другим процессом: 'pyglossary-clone\\.git\\objects\\pack\\tmp_pack_EMkDJk'

(Originally I reported this on Poetry): This is because the project folder contains a symbolic link and the ususual git installation for Windows does not try to create them by default, unlike dulwich. To quote my post:

The reason seems to be that by default git for windows does not create real symlinks, but instead some sort of text files. Windows apparently does not like non-admin users having the right to create symlinks because they enable symlink attacks. I don't know how much of a problem they are, maybe MS also don't like potential bugs because symlinks are less stable than on Linux.

The problem can apparently be fixed by enabling Windows developer mode, which allows symlinks even for non-admin users

I think that most developers on Windows don't know that this developer mode even exists and don't have it enabled and executing poetry with admin rights does not seem like a good solution (correct me if I'm wrong).

I think it would be great to have a better error message here that says that you don't have permissions to create symlinks and that you can enable Windows developer mode to make it work. Currently it is not that easy to figure out what is wrong.

[jelmer]

Thanks for the report; I'm open to a PR improving the error message.

[Vuizur]

I tried adding a small message on my fork here: https://github.com/Vuizur/dulwich

Although unfortunately I tried testing it and couldn't build wheels on windows for some reason when installing from Git, so I don't know if it works. (I also tried installing the build tools that are linked in the error.)

 C:\Users\hanne\Documents\Programme\dulwich-test-2> poetry add git+https://github.com/Vuizur/dulwich

Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing dulwich (0.20.45 1b4080d)

  CalledProcessError

  Command 'C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-2-YdXoDw1F-py3.10\Scripts\python.exe C:\Users\hanne\AppData\Roaming\pypoetry\venv\lib\site-packages\virtualenv\seed\wheels\embed\pip-22.2.2-py3-none-any.whl\pip install --use-pep517 --disable-pip-version-check --prefix C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-2-YdXoDw1F-py3.10 --upgrade --no-deps C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-2-YdXoDw1F-py3.10\src\dulwich' returned non-zero exit status 1.

  at C:\Python310\lib\subprocess.py:524 in run
       520│             # We don't call process.wait() as .__exit__ does that for us.
       521│             raise
       522│         retcode = process.poll()
       523│         if check and retcode:
    →  524│             raise CalledProcessError(retcode, process.args,
       525│                                      output=stdout, stderr=stderr)      
       526│     return CompletedProcess(process.args, retcode, stdout, stderr)      
       527│
       528│

The following error occurred when trying to handle this error:


  EnvCommandError

  Command C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-2-YdXoDw1F-py3.10\Scripts\python.exe C:\Users\hanne\AppData\Roaming\pypoetry\venv\lib\site-packages\virtualenv\seed\wheels\embed\pip-22.2.2-py3-none-any.whl\pip install --use-pep517 --disable-pip-version-check --prefix C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-2-YdXoDw1F-py3.10 --upgrade --no-deps C:\Users\hanne\AppData\Local\pypoetry\Cache\virtualenvs\dulwich-test-2-YdXoDw1F-py3.10\src\dulwich 
errored with the following return code 1, and output:
  Processing c:\users\hanne\appdata\local\pypoetry\cache\virtualenvs\dulwich-test-2-ydxodw1f-py3.10\src\dulwich
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Building wheels for collected packages: dulwich
    Building wheel for dulwich (pyproject.toml): started
    Building wheel for dulwich (pyproject.toml): finished with status 'error'       
    error: subprocess-exited-with-error

    Building wheel for dulwich (pyproject.toml) did not run successfully.
    exit code: 1

    [19 lines of output]
    running bdist_wheel
    running build
    running build_py
    copying dulwich\index.py -> build\lib.win-amd64-cpython-310\dulwich
    running egg_info
    writing dulwich.egg-info\PKG-INFO
    writing dependency_links to dulwich.egg-info\dependency_links.txt
    writing entry points to dulwich.egg-info\entry_points.txt
    writing requirements to dulwich.egg-info\requires.txt
    writing top-level names to dulwich.egg-info\top_level.txt
    reading manifest file 'dulwich.egg-info\SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no directories found matching 'dulwich\tests\data'
    adding license file 'COPYING'
    adding license file 'AUTHORS'
    writing manifest file 'dulwich.egg-info\SOURCES.txt'
    running build_ext
    building 'dulwich._objects' extension
    error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft 
C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/        
    [end of output]

    note: This error originates from a subprocess, and is likely not a problem with 
pip.
    ERROR: Failed building wheel for dulwich
  Failed to build dulwich
  ERROR: Could not build wheels for dulwich, which is required to install pyproject.toml-based projects


  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\env.py:1473 in _run
      1469│                 output = subprocess.check_output(
      1470│                     command, stderr=subprocess.STDOUT, env=env, **kwargs      1471│                 )
      1472│         except CalledProcessError as e:
    → 1473│             raise EnvCommandError(e, input=input_)
      1474│
      1475│         return decode(output)
      1476│
      1477│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:      

The following error occurred when trying to handle this error:


  PoetryException

  Failed to install C:/Users/hanne/AppData/Local/pypoetry/Cache/virtualenvs/dulwich-test-2-YdXoDw1F-py3.10/src/dulwich

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\poetry\utils\pip.py:51 in pip_install
       47│ 
       48│     try:
       49│         return environment.run_pip(*args)
       50│     except EnvCommandError as e:
    →  51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       52│

[jelmer]

I tried adding a small message on my fork here: https://github.com/Vuizur/dulwich

Although unfortunately I tried testing it and couldn't build wheels on windows for some reason when installing from Git, so I don't know if it works. (I also tried installing the build tools that are linked in the error.)

This appears to be happening because you don't have a C environment installed for building the C extensions.

I've left some comments on the branch.

[DanielW10001]

I am having the same problem here when using Poetry, which at the time uses [email protected].

[jelmer]

I'm still open to PRs that improve the error message. Alternatively, we could allow passing in another function to be called for creating symlinks - so you could e.g. create regular files instead.