Concurrent cache access causes errors on Windows

[charliermarsh]

See the comments in https://github.com/astral-sh/uv/issues/7434#issuecomment-2615798114.

[petamas]

@charliermarsh: I have some bad news - there are some concurrency issues still present in uv 0.6.13.

Could you reopen this issue and take another look?

Repro steps

1. Put the following in repro.bat:

@echo off
setlocal

set UV_CACHE_DIR=%CD%\cache
set UV_PYTHON_INSTALL_DIR=%CD%\python

if exist "%UV_CACHE_DIR%" rmdir /S /Q "%UV_CACHE_DIR%"
if exist "%UV_PYTHON_INSTALL_DIR%" rmdir /S /Q "%UV_PYTHON_INSTALL_DIR%"
if exist *.log del *.log

for /L %%i in (1,1,20) do start cmd /c "uv run hello.py >%%i.log 2>&1"

2. Put the following in hello.py:

# /// script
# requires-python = ">=3.12"
# dependencies = [
#   "oslex==0.1.3",
#   "comtypes==1.4.9",
# ]
# ///

print("Hello, world!")

3. Run repro.bat

Errors

In the generated log files, the following errors can be observed:

error: Failed to install: comtypes-1.4.9-py3-none-any.whl (comtypes==1.4.9)
  Caused by: RECORD file is invalid
  Caused by: The process cannot access the file because it is being used by another process. (os error 32)

and

error: Failed to install: comtypes-1.4.9-py3-none-any.whl (comtypes==1.4.9)
  Caused by: failed to copy file from d:\code\experiments\uv-parallel-issue\cache\archive-v0\Tn2MVVxErMR6qBVuAcGGi\comtypes-1.4.9.dist-info\RECORD to d:\code\experiments\uv-parallel-issue\cache\environments-v2\hello-8b8d8853a364cdcc\Lib\site-packages\comtypes-1.4.9.dist-info\RECORD: The process cannot access the file because it is being used by another process. (os error 32)

The number of failed runs vary, as does the exact wheel that fails.

Notes

Note1: I haven't ran into these exact errors with earlier uv versions; 0.4.15 usually failed with issues related to .rkyv files, while 0.5.26 failes with "The file or directory is not a reparse point. (os error 4390)" or "The process cannot access the file because it is being used by another process. (os error 32)" (without additional details).

Note2: I tried reproducing using the Powershell script in https://github.com/astral-sh/uv/pull/11007, but I couldn't reproduce the issue that way. I don't know whether the key difference lies between uv run and uv pip install, or something else entirely.

[petamas]

I took a peek under the hood. I think the issue might be happening within this call to synchronized_copy(): https://github.com/astral-sh/uv/blob/3ece75fc123088bf7f29ff44a7ae1837271c93e0/crates/uv-install-wheel/src/linker.rs#L303

I don't really know Rust, and I'm not familiar with the locking policies within uv, but is it possible that synchronized_copy() should be extended with locking the from directory in addition to the to directory? (My theory is that the source RECORD file gets written to from one process while the other is trying to copy it. I assume that writing to the source directory, which is in the archive cache, is already lock-protected.)

[zanieb]

cc @Gankra who has touched the cache recently

[majutsushi]

I just ran into this while trying to get a colleague set up who was using Windows. Trying to install any moderately complex library would result in the "file is used by another process" error. We had to resort to using pip to get things unblocked.

[petamas]

@majutsushi: Which uv version are you using?

~I can no longer reproduce the issue with uv 0.7.11, it might've been fixed as a side effect of some other change.~

~Edit: Strike that, I can't seem to reproduce it using uv 0.6.13 either, even though I was able to a couple hours ago.~

Edit: see next comment.

[petamas]

Managed to reproduce in Windows Sandbox using uv 0.7.11 with these exact repro steps: https://github.com/astral-sh/uv/issues/11002#issuecomment-2787658535