`pdm add tables` fails to create symlinks

[wikiped]

• [x] I have searched the issue tracker and believe that this is not a duplicate.

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

On windows using powershell prompt:

pushd $env:TEMP
mkdir test-package
cd test-package
pdm init -n -p .
pdm add numpy cython blosc numexpr tables
pdm run python -c "import tables"

Note output and then clean-up:

popd
rm $env:TEMP\test-package -R

Actual behavior

pdm add emits couple of error messages:

Adding packages to default dependencies: numpy, cython, blosc, numexpr, tables
🔒 Lock successful
Changes are written to pdm.lock.
Changes are written to pyproject.toml.
Synchronizing working set with lock file: 7 to add, 0 to update, 0 to remove

  ✔ Install numexpr 2.8.3 successful
  ✔ Install packaging 21.3 successful
  ✔ Install pyparsing 3.0.9 successful
  ✔ Install cython 0.29.32 successful
  ✖ Install tables 3.7.0 failed
  ✔ Install numpy 1.23.3 successful
  ✖ Install blosc 1.10.6 failed
Retry failed jobs
  ✖ Install blosc 1.10.6 failed
  ✖ Install tables 3.7.0 failed

ERRORS:
add blosc failed:
Traceback (most recent call last):
...
  File "D:\Python\.pdm\venv\lib\site-packages\pdm\installers\installers.py", line 89, in
_create_symlinks_recursively
    os.remove(destination_root)
PermissionError: [WinError 5] Access is denied: 'D:\\Temp\\test-package\\__pypackages__\\3.10\\lib\\pkgconfig'
add tables failed:
Traceback (most recent call last):
...
  File "D:\Python\.pdm\venv\lib\site-packages\pdm\installers\installers.py", line 89, in
_create_symlinks_recursively
    os.remove(destination_root)
PermissionError: [WinError 5] Access is denied: 'D:\\Temp\\test-package\\__pypackages__\\3.10\\lib\\site-packages'

See C:\Users\User\AppData\Local\Temp\pdm-install-egmqjmw4.log for detailed debug log.
[InstallationError]: Some package operations are not complete yet

But the errors are not indicating the real problem (or retry by pdm has helped to mitigate them) since manual inspection of both locations reported by error messages shows that pdm add did actually COPIED (not symlinked - which perhaps deserves its own issue) dll files from the cache.

One of the errors above was about blosc. It has somewhat similar structure to tables module (some pre-compiled dlls going here and there). But it was installed (symlinks created / files copied) successfully, which can be tested with:

pdm run python -c "import blosc"

Which does not output any errors.

Output from pdm run python -c "import tables" command:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'tables'

Expected behavior

There should be NO output if import was successful.

Or the output will be from tables module itself about missing libraries - which indicates that import was successful.

Environment Information

> pdm info

# pdm info 
PDM version:
  2.1.4
Python Interpreter:
  D:\Python\Python310\python.EXE (3.10)
Project Root:
  D:/Python/.pdm/global-project
Project Packages:
  None

> pdm info --env

{
  "implementation_name": "cpython",
  "implementation_version": "3.10.5",
  "os_name": "nt",
  "platform_machine": "AMD64",
  "platform_release": "10",
  "platform_system": "Windows",
  "platform_version": "10.0.19043",
  "python_full_version": "3.10.5",
  "platform_python_implementation": "CPython",
  "python_version": "3.10",
  "sys_platform": "win32"
}

Manually creating symlinks from the cached tables-3.7.0-cp310-cp310-win_amd64 directory to lib and others in __packages__ fixes the problem.

[frostming]

Clear the caches and __pypackages__ and try reinstall Or disable the install.cache and reinstall to see if the error still persists. At least I can't reproduce locally, on Windows, too.

[wikiped]

If install.cache is false in config - then no errors are appearing.

But with cache enabled (and therefore symlinks being used) I am not sure how it would be possible for this installation to go without errors.

I did some debugging and here is simplified printout from executing command that creates symlinks:

In pdm\installers\installers.py, from line 89 in _create_symlinks_recursively:

source: <PDM_HOME>\Cache\packages\tables-3.7.0-cp310-cp310-win_amd64\lib
destination: <PACKAGE_ROOT>\.venv\Lib\site-packages

root: <PDM_HOME>\Cache\packages\tables-3.7.0-cp310-cp310-win_amd64\lib
relpath: .
is_namespace_package(root): True
destination_root: <PACKAGE_ROOT>\.venv\Lib\site-packages\.

  root: <PDM_HOME>\Cache\packages\tables-3.7.0-cp310-cp310-win_amd64\lib\site-packages
  relpath: site-packages
  is_namespace_package(root): False
  destination_root: <PACKAGE_ROOT>\.venv\Lib\site-packages\site-packages
  os.symlink(
    <PDM_HOME>\Cache\packages\tables-3.7.0-cp310-cp310-win_amd64\lib\site-packages,
    <PACKAGE_ROOT>\.venv\Lib\site-packages\site-packages,
    True)

  root: <PDM_HOME>\Cache\packages\tables-3.7.0-cp310-cp310-win_amd64\lib\tables
  relpath: tables
  is_namespace_package(root): False
  destination_root: <PACKAGE_ROOT>\.venv\Lib\site-packages\tables
  Failed command: os.remove(<PACKAGE_ROOT>\.venv\Lib\site-packages\tables)
  Content of destination_root:
    blosc.dll
    bzip2.dll
    hdf5.dll
    zlib.dll

So by the time command gets to lib\tables in the wheel it finds that the destination already exists and when it tries to remove it with os.remove(...) it gets PermissionError because the directory is not empty. And the reason it is not empty is because before this command pdm\installers\installers.py" at line 279 calls _install_wheel, which in its turn calls to the following part in installer_core.py, line 95, in install` command:

# Write all the files from the wheel.
    for record_elements, stream, is_executable in source.get_contents():
    ...
        record = destination.write_file(...)  <-- This line writes files that create an error in os.remove(...) above

And if we print the content of the source:

> print([c[0][0] for c in source.get_contents()])

The files that are later creating problems are being copied here:

['tables-3.7.0.data/data/Lib/site-packages/tables/blosc.dll',
'tables-3.7.0.data/data/Lib/site-packages/tables/bzip2.dll',
'tables-3.7.0.data/data/Lib/site-packages/tables/hdf5.dll',
'tables-3.7.0.data/data/Lib/site-packages/tables/zlib.dll',
'tables-3.7.0.dist-info/DELVEWHEEL',
'tables-3.7.0.dist-info/entry_points.txt',
'tables-3.7.0.dist-info/LICENSE.txt',
'tables-3.7.0.dist-info/METADATA',
'tables-3.7.0.dist-info/RECORD',
'tables-3.7.0.dist-info/top_level.txt',
'tables-3.7.0.dist-info/WHEEL']

I am not sure if the installer is doing what it is supposed to do here or not, but assuming it works correctly one way to solve the problem would be to use shutil.rmtree, which does not raise if the folder is not empty.

It might well be that this problem is specific to tables package, which has this "redundant" site-packages sub-folder containing duplicates of the files that are anyway present in tables folder in the root of the wheel.

I can do a PR with shutil.rmtree to fix this issue, if this is of any help.

[eltoder]

@wikiped @frostming I have potentially the same issue, which I can reproduce very reliably. It only happens with install.cache=true, so I had to disable caching.

Steps to reproduce

mkdir proj && cd proj
mkdir __pypackages__  # force PEP 582 mode
pdm init -n
pdm config -l install.cache true  # enable caching
pdm add pyre-check

Fails with an error:

ERRORS:
add pyre-check failed:
Traceback (most recent call last):
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/elt/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/installers/synchronizers.py", line 236, in
install_candidate
    self.manager.install(can)
  File "/home/elt/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/installers/manager.py", line 39, in install
    installer(str(prepared.build()), self.environment, prepared.direct_url())
  File "/home/elt/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/installers/installers.py", line 254, in
install_wheel_with_cache
    dist_info_dir = _install_wheel(
  File "/home/elt/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/installers/installers.py", line 281, in
_install_wheel
    install(source, destination, additional_metadata=additional_metadata or {})
  File "/home/elt/.local/share/pdm/venv/lib/python3.10/site-packages/installer/_core.py", line 131, in install
    destination.finalize_installation(
  File "/home/elt/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/installers/installers.py", line 162, in
finalize_installation
    for relpath in _create_symlinks_recursively(
  File "/home/elt/.local/share/pdm/venv/lib/python3.10/site-packages/pdm/installers/installers.py", line 90, in
_create_symlinks_recursively
    os.remove(destination_root)
IsADirectoryError: [Errno 21] Is a directory: '/home/elt/code/pdm/__pypackages__/3.10/lib/pyre_check'

The same works fine without caching.

Environment Information:

PDM version:
  2.4.2
Python Interpreter:
  /usr/bin/python (3.10)
Project Root:
  /home/elt/code/pdm
Project Packages:
  /home/elt/code/pdm/__pypackages__/3.10
{
  "implementation_name": "cpython",
  "implementation_version": "3.10.6",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "5.15.79.1-microsoft-standard-WSL2",
  "platform_system": "Linux",
  "platform_version": "#1 SMP Wed Nov 23 01:01:46 UTC 2022",
  "python_full_version": "3.10.6",
  "platform_python_implementation": "CPython",
  "python_version": "3.10",
  "sys_platform": "linux"
}

[frostming]

Try with the new version of PDM

[wikiped]

@frostming Thank you very much for fixing this! No errors with pdm add tables and all .dll files are being symlinked properly now.