dvc
dvc copied to clipboard
Published
20 hours ago •
iterative
iterative
dvc pull: fails with `PermissionError: [WinError 32]`
Bug Report
(Rehash of comments from https://github.com/iterative/dvc/issues/6403#issuecomment-1642817641 and following).
Comment https://github.com/iterative/dvc/issues/9749#issuecomment-1646796239 contains instructions to reproduce the WinError 32 problems from scratch.
Description
On Windows in a DVC project on the console while editing this project in VSCode.
(base) PS C:\Users\achimgaedke\Code\xxx> dvc pull
ERROR: unexpected error - [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\achimgaedke\\Code\\xxx\\train-out\\model_19.pth'
I would expect DVC to pull in my models from S3.
A successful pull looks like this:
(base) PS C:\Users\achimgaedke\Code\xxx> $Env:GIT_OPTIONAL_LOCKS=0
(base) PS C:\Users\achimgaedke\Code\xxx> dvc pull
M train-out\
1 file modified
(base) PS C:\Users\achimgaedke\Code\xxx>
Reproduce
For me, it reproduces like this (intermittently though):
- Start working on my project in VSCode
- Delete the model files in train-out
- Open the terminal
- Run
dvc pull
The 20ish files pulled are each 300MB. Later (once I have more time) I'll try to create a sharable project. (Yeah, it is hard to report bugs from ongoing work).
Debug information
Note the line:
2023-07-20 14:39:20,177 DEBUG: failed to create 'C:\Users\achimgaedke\Code\xxx_c14_p113\train-out\model_4.pth' from 'C:\Users\achimgaedke\Code\xxx_c14_p113\.dvc\cache2\files\md5\c9\e8b9140bc7187529a5fcb33fe29702' - [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\achimgaedke\\Code\\xxx_c14_p113\\train-out\\.PLs2vrX8fXNan9LjzfiH3V.tmp' -> 'C:\\Users\\achimgaedke\\Code\\xxx_c14_p113\\train-out\\model_4.pth'
It looks like the creation of the file in the target directory train-out from the cache directory fails (i.e. the caching itself works).
- I've disabled the virus scan and indexer for the Code directory (following the instructions at https://dvc.org/doc/user-guide/how-to/run-dvc-on-windows)
- I am running VSCode with the git and dvc extensions - git status might interfere.
- I've installed dvc, git and VSCode with scoop.
- I have enabled long pathname and symlink support in windows (following the advice of
scoop checkup).
Which process would be so nosy to pick up new file immediately? Maybe git status if this dir/file is not listed in gitignore?)
In my experience this error does not appear when setting $Env:GIT_OPTIONAL_LOCKS=0. (see https://git-scm.com/docs/git#Documentation/git.txt-codeGITOPTIONALLOCKScode).
Environment information
Output of dvc doctor:
dvc doctor
DVC version: 3.5.1
------------------
Platform: Python 3.10.11 on Windows-10-10.0.22621-SP0
Subprojects:
Supports:
azure (adlfs = 2023.4.0, knack = 0.10.1, azure-identity = 1.13.0),
gdrive (pydrive2 = 1.16.1),
gs (gcsfs = 2023.6.0),
hdfs (fsspec = 2023.6.0, pyarrow = 12.0.1),
http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
oss (ossfs = 2021.8.0),
s3 (s3fs = 2023.6.0, boto3 = 1.26.161),
ssh (sshfs = 2023.4.1),
webdav (webdav4 = 0.9.8),
webdavs (webdav4 = 0.9.8),
webhdfs (fsspec = 2023.6.0)
Config:
Global: C:\Users\achimgaedke\AppData\Local\iterative\dvc
System: C:\ProgramData\iterative\dvc
Cache types: hardlink, symlink
Cache directory: NTFS on C:\
Caches: local
Remotes: s3
Workspace directory: NTFS on C:\
Repo: dvc, git
Repo.site_cache_dir: C:\ProgramData\iterative\dvc\Cache\repo\a5786f5395aa8da08287147b62b67b3b
scoop list
Installed apps:
Name Version Source Updated Info
---- ------- ------ ------- ----
7zip 23.01 main 2023-06-28 15:33:15
aws 2.13.2 main 2023-07-20 08:59:51
aws-vault 7.2.0 main 2023-06-23 09:17:31
dark 3.11.2 main 2023-06-12 08:34:04
dbeaver 23.1.2 extras 2023-07-10 08:48:20
dvc 3.5.1 main 2023-07-12 07:30:56
firefox 115.0.2 extras 2023-07-12 07:30:03
gh 2.32.0 main 2023-07-12 13:12:19
git 2.41.0.3 main 2023-07-18 20:50:20
innounp 0.50 main 2023-07-07 09:19:53
keepass 2.54 extras 2023-06-20 10:34:13
mambaforge 23.1.0-4 extras 2023-07-17 08:10:19
sourcetree 3.4.14 extras 2023-06-29 08:06:09
sudo 0.2020.01.26 main 2023-07-11 07:30:39
terraform 1.5.3 main 2023-07-13 15:37:19
vscode 1.80.1 extras 2023-07-17 08:10:59
Just updated to version 3.6, debug output looks different in the details, result is the same...
Looks like it downloaded it successfully, but it is checkout (pull actually equals fetch + checkout) that is failing.
I see that you are using symlink/hardlink, that might be what is triggering it. Try using copy, just so we can narrow it down. Also if your files are not that big, maybe copy would be acceptable for you overall, since it is safer than symlink/hardlink, esp on windows where it has more nuances.
as requested...
dvc config cache.type copy
dvc checkout --relink
and then deleted the directory and issued dvc pull -v.
The reporting about the cache type seems misleading or the config isn't read. The config file and dvc config cache.type both state copy.
Another way of avoiding this ordeal is to close VSCode and only run the commands on the terminal... Defeats the purpose of VSCode though....
> dvc config --list
core.analytics=false
remote.aws-storage.url=s3://xxxx/dvc-storage/
core.remote=aws-storage
core.autostage=true
cache.type=copy
All right, I've created a new project to reproduce similar (or the same) trouble:
Copy the python program create_data_dir.py into an empty directory (e.g. dvc_stress_test):
import pathlib
import random
import shutil
def create_big_random_file(seed: int, path: pathlib.Path, mbsize: int):
mbibyte = 1 << 20
random_data = random.Random(seed).randbytes(mbibyte)
with path.open(mode="wb") as f:
for _ in range(mbsize):
f.write(random_data)
data_dir = pathlib.Path("data")
shutil.rmtree(data_dir, ignore_errors=True)
data_dir.mkdir()
for file_idx in range(10):
create_big_random_file(
file_idx + 42, data_dir / f"big_random_file{file_idx:02d}", 1000
)
and then execute:
git init
dvc init
dvc config cache.type copy
dvc checkout --relink
dvc stage add -n create_data -d create_data_dir.py -o data python create_data_dir.py
dvc repro -v
Do this:
- in the PowerShell (outside VSCode) - works fine
- using VSCode with the (git/DVC extensions installed) - this fails, the output is copied below:
PS C:\Users\achimgaedke\extCode\dvc_stress_test> dvc init
>> dvc config cache.type copy
>> dvc checkout --relink
>>
>> dvc stage add -n create_data -d create_data_dir.py -o data python create_data_dir.py
>> dvc repro -v
Initialized DVC repository.
You can now commit the changes to git.
What's next?
------------
- Check out the documentation: <https://dvc.org/doc>
- Get help and share ideas: <https://dvc.org/chat>
- Star us on GitHub: <https://github.com/iterative/dvc>
Relinked successfully
Added stage 'create_data' in 'dvc.yaml'
To track the changes with git, run:
git add dvc.yaml
To enable auto staging, run:
dvc config core.autostage true
2023-07-23 21:35:55,624 DEBUG: v3.7.0, CPython 3.10.11 on Windows-10-10.0.22621-SP0
2023-07-23 21:35:55,625 DEBUG: command: repro -v
2023-07-23 21:35:56,018 DEBUG: Lockfile for 'dvc.yaml' not found
2023-07-23 21:35:56,028 DEBUG: Lockfile for 'dvc.yaml' not found
2023-07-23 21:35:56,152 DEBUG: Dependency 'create_data_dir.py' of stage: 'create_data' changed because it is 'modified'.
2023-07-23 21:35:56,154 DEBUG: stage: 'create_data' changed.
2023-07-23 21:35:56,157 DEBUG: Removing output 'data' of stage: 'create_data'.
2023-07-23 21:35:56,158 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data'
2023-07-23 21:35:56,165 DEBUG: {'create_data_dir.py': 'modified'}
Running stage 'create_data':
> python create_data_dir.py
2023-07-23 21:36:04,216 DEBUG: Lockfile for 'dvc.yaml' not found
2023-07-23 21:36:17,502 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-07-23 21:36:17,504 DEBUG: Computed stage: 'create_data' md5: 'f245734c5a8cc0588b72210046fab8da'
2023-07-23 21:36:17,526 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-07-23 21:36:17,529 DEBUG: Preparing to transfer data from 'memory://dvc-staging-md5/60dc2168a9f5c260e9d6f8ab6a80dee5e8805025a4f7d1335e0d78f177eb17a1' to 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-07-23 21:36:17,530 DEBUG: Preparing to collect status from 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-07-23 21:36:17,531 DEBUG: Collecting status from 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-07-23 21:36:17,538 DEBUG: Preparing to collect status from 'memory://dvc-staging-md5/60dc2168a9f5c260e9d6f8ab6a80dee5e8805025a4f7d1335e0d78f177eb17a1'
2023-07-23 21:36:17,539 DEBUG: transfer dir: md5: 625be7c4ce6f1264a616c17b0e648bdc.dir with 10 files
2023-07-23 21:36:17,556 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-07-23 21:36:17,563 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\.DRmFuaMqh6KghWvzF29XLz.tmp'
2023-07-23 21:36:17,564 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5\.cmZL9BuWGVHrxL6Rr2tyNz.tmp'
2023-07-23 21:36:17,567 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file04'
2023-07-23 21:36:18,336 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file06'
2023-07-23 21:36:19,146 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file09'
2023-07-23 21:36:19,150 ERROR: failed to reproduce 'create_data': [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\achimgaedke\\extCode\\dvc_stress_test\\data\\big_random_file09'
Traceback (most recent call last):
File "dvc_objects\fs\utils.py", line 89, in _unlink
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\achimgaedke\\extCode\\dvc_stress_test\\data\\big_random_file09'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "dvc\repo\reproduce.py", line 199, in _reproduce
File "dvc\repo\reproduce.py", line 129, in _reproduce_stage
File "funcy\decorators.py", line 47, in wrapper
File "dvc\stage\decorators.py", line 43, in rwlocked
File "funcy\decorators.py", line 68, in __call__
File "dvc\stage\__init__.py", line 433, in reproduce
File "dvc\stage\decorators.py", line 43, in rwlocked
File "funcy\decorators.py", line 68, in __call__
File "dvc\stage\__init__.py", line 626, in run
File "funcy\decorators.py", line 47, in wrapper
File "dvc\stage\decorators.py", line 43, in rwlocked
File "funcy\decorators.py", line 68, in __call__
File "dvc\stage\__init__.py", line 549, in commit
File "dvc\output.py", line 793, in commit
File "dvc\output.py", line 752, in _checkout
File "dvc_data\hashfile\checkout.py", line 276, in checkout
File "dvc_data\hashfile\checkout.py", line 223, in _checkout
File "dvc_data\hashfile\checkout.py", line 88, in _checkout_file
File "dvc_data\hashfile\checkout.py", line 59, in _relink
File "dvc_data\hashfile\checkout.py", line 55, in _remove
File "dvc_objects\fs\base.py", line 470, in rm
File "dvc_objects\fs\local.py", line 119, in rm
File "dvc_objects\fs\utils.py", line 101, in remove
File "dvc_objects\fs\utils.py", line 91, in _unlink
File "dvc_objects\fs\utils.py", line 84, in _chmod
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\achimgaedke\\extCode\\dvc_stress_test\\data\\big_random_file09'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "dvc\cli\__init__.py", line 209, in main
File "dvc\cli\command.py", line 26, in do_run
File "dvc\commands\repro.py", line 13, in run
File "dvc\repo\__init__.py", line 64, in wrapper
File "dvc\repo\scm_context.py", line 151, in run
File "dvc\repo\reproduce.py", line 260, in reproduce
File "dvc\repo\reproduce.py", line 203, in _reproduce
File "dvc\repo\reproduce.py", line 167, in _raise_error
dvc.exceptions.ReproductionError: failed to reproduce 'create_data'
2023-07-23 21:36:19,160 DEBUG: Analytics is disabled.
PS C:\Users\achimgaedke\extCode\dvc_stress_test>
Here some diagnostics
dvc config --list
core.analytics=false
cache.type=copy
dvc doctor
DVC version: 3.7.0
------------------
Platform: Python 3.10.11 on Windows-10-10.0.22621-SP0
Subprojects:
Supports:
azure (adlfs = 2023.4.0, knack = 0.10.1, azure-identity = 1.13.0),
gdrive (pydrive2 = 1.16.1),
gs (gcsfs = 2023.6.0),
hdfs (fsspec = 2023.6.0, pyarrow = 12.0.1),
http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
oss (ossfs = 2021.8.0),
s3 (s3fs = 2023.6.0, boto3 = 1.26.161),
ssh (sshfs = 2023.7.0),
webdav (webdav4 = 0.9.8),
webdavs (webdav4 = 0.9.8),
webhdfs (fsspec = 2023.6.0)
Config:
Global: C:\Users\achimgaedke\AppData\Local\iterative\dvc
System: C:\ProgramData\iterative\dvc
Cache types: hardlink, symlink
Cache directory: NTFS on C:\
Caches: local
Remotes: None
Workspace directory: NTFS on C:\
Repo: dvc, git
Repo.site_cache_dir: C:\ProgramData\iterative\dvc\Cache\repo\c9f823bbd557bbd5b830efcb5e947c49
The harddrive has at the time 116GB free space (yeah, I cleared out some of my "real files" >>10GB).
(ah, more trouble can be experienced with a repeat dvc repro or dvc destroy after the first dvc repro failed.)
@AchimGaedkeLynker Have you been able to determine if GIT_OPTIONAL_LOCKS makes a difference?
@dberenbaum - no it doesn't seem to help. The error message also changed with the latest version/s:
PS C:\Users\achimgaedke\extCode\dvc_stress_test> $Env:GIT_OPTIONAL_LOCKS=0
PS C:\Users\achimgaedke\extCode\dvc_stress_test> dvc repro -v
2023-08-05 16:00:55,168 DEBUG: v3.11.1, CPython 3.10.11 on Windows-10-10.0.22621-SP0
2023-08-05 16:00:55,168 DEBUG: command: repro -v
2023-08-05 16:00:55,431 DEBUG: Lockfile for 'dvc.yaml' not found
2023-08-05 16:00:55,443 DEBUG: Lockfile for 'dvc.yaml' not found
2023-08-05 16:00:55,543 DEBUG: Dependency 'create_data_dir.py' of stage: 'create_data' changed because it is 'modified'.
2023-08-05 16:00:55,543 DEBUG: stage: 'create_data' changed.
2023-08-05 16:00:55,545 DEBUG: Removing output 'data' of stage: 'create_data'.
2023-08-05 16:00:55,546 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data'
2023-08-05 16:00:55,549 DEBUG: {'create_data_dir.py': 'modified'}
Running stage 'create_data':
> python create_data_dir.py
2023-08-05 16:01:03,181 DEBUG: Lockfile for 'dvc.yaml' not found
2023-08-05 16:01:15,126 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-08-05 16:01:15,129 DEBUG: Computed stage: 'create_data' md5: '26237bb165f9c74df6920096edc70031'
2023-08-05 16:01:15,139 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-08-05 16:01:15,142 DEBUG: Preparing to transfer data from 'memory://dvc-staging-md5/60dc2168a9f5c260e9d6f8ab6a80dee5e8805025a4f7d1335e0d78f177eb17a1' to 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-08-05 16:01:15,143 DEBUG: Preparing to collect status from 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-08-05 16:01:15,143 DEBUG: Collecting status from 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-08-05 16:01:15,146 DEBUG: Preparing to collect status from 'memory://dvc-staging-md5/60dc2168a9f5c260e9d6f8ab6a80dee5e8805025a4f7d1335e0d78f177eb17a1'
2023-08-05 16:01:15,148 DEBUG: transfer dir: md5: 625be7c4ce6f1264a616c17b0e648bdc.dir with 10 files
2023-08-05 16:01:15,179 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-08-05 16:01:15,185 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\.C3iskTZZHBAtgstpgrxdsh.tmp'
2023-08-05 16:01:15,187 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5\.BKbBCL5G3EqF9RMsrbggke.tmp'
2023-08-05 16:01:15,191 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file08'
2023-08-05 16:01:16,086 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file05'
2023-08-05 16:01:17,202 ERROR: failed to reproduce 'create_data': No possible cache link types for 'C:\Users\achimgaedke\extCode\dvc_stress_test\data'. See <https://dvc.org/doc/user-guide/troubleshooting#cache-types> for more information.
Traceback (most recent call last):
File "dvc\repo\reproduce.py", line 199, in _reproduce
File "dvc\repo\reproduce.py", line 129, in _reproduce_stage
File "funcy\decorators.py", line 47, in wrapper
File "dvc\stage\decorators.py", line 43, in rwlocked
File "funcy\decorators.py", line 68, in __call__
File "dvc\stage\__init__.py", line 433, in reproduce
File "funcy\decorators.py", line 47, in wrapper
File "dvc\stage\decorators.py", line 43, in rwlocked
File "funcy\decorators.py", line 68, in __call__
File "dvc\stage\__init__.py", line 626, in run
File "funcy\decorators.py", line 47, in wrapper
File "dvc\stage\decorators.py", line 43, in rwlocked
File "funcy\decorators.py", line 68, in __call__
File "dvc\stage\__init__.py", line 556, in commit
dvc.exceptions.CacheLinkError: No possible cache link types for 'C:\Users\achimgaedke\extCode\dvc_stress_test\data'. See <https://dvc.org/doc/user-guide/troubleshooting#cache-types> for more information.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "dvc\cli\__init__.py", line 209, in main
File "dvc\cli\command.py", line 26, in do_run
File "dvc\commands\repro.py", line 13, in run
File "dvc\repo\__init__.py", line 64, in wrapper
File "dvc\repo\scm_context.py", line 151, in run
File "dvc\repo\reproduce.py", line 260, in reproduce
File "dvc\repo\reproduce.py", line 203, in _reproduce
File "dvc\repo\reproduce.py", line 167, in _raise_error
dvc.exceptions.ReproductionError: failed to reproduce 'create_data'
2023-08-05 16:01:17,208 DEBUG: Analytics is disabled.
PS C:\Users\achimgaedke\extCode\dvc_stress_test>
Here's a successful run outside VSCode
PS C:\Users\achimgaedke\extCode\dvc_stress_test> dvc repro -v
2023-08-05 16:11:47,283 DEBUG: v3.11.1, CPython 3.10.11 on Windows-10-10.0.22621-SP0
2023-08-05 16:11:47,283 DEBUG: command: repro -v
2023-08-05 16:11:47,543 DEBUG: Lockfile for 'dvc.yaml' not found
2023-08-05 16:11:47,543 DEBUG: Lockfile for 'dvc.yaml' not found
2023-08-05 16:11:47,625 DEBUG: Dependency 'create_data_dir.py' of stage: 'create_data' changed because it is 'modified'.
2023-08-05 16:11:47,634 DEBUG: stage: 'create_data' changed.
2023-08-05 16:11:47,635 DEBUG: Removing output 'data' of stage: 'create_data'.
2023-08-05 16:11:47,635 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data'
2023-08-05 16:11:47,639 DEBUG: {'create_data_dir.py': 'modified'}
Running stage 'create_data':
> python create_data_dir.py
2023-08-05 16:11:54,435 DEBUG: Lockfile for 'dvc.yaml' not found
2023-08-05 16:12:10,717 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-08-05 16:12:10,717 DEBUG: Computed stage: 'create_data' md5: '26237bb165f9c74df6920096edc70031'
2023-08-05 16:12:10,748 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-08-05 16:12:10,764 DEBUG: Preparing to transfer data from 'memory://dvc-staging-md5/60dc2168a9f5c260e9d6f8ab6a80dee5e8805025a4f7d1335e0d78f177eb17a1' to 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-08-05 16:12:10,764 DEBUG: Preparing to collect status from 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-08-05 16:12:10,764 DEBUG: Collecting status from 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5'
2023-08-05 16:12:10,768 DEBUG: Preparing to collect status from 'memory://dvc-staging-md5/60dc2168a9f5c260e9d6f8ab6a80dee5e8805025a4f7d1335e0d78f177eb17a1'
2023-08-05 16:12:10,768 DEBUG: transfer dir: md5: 625be7c4ce6f1264a616c17b0e648bdc.dir with 10 files
2023-08-05 16:12:10,780 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-08-05 16:12:10,780 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\.Vx4YGUwbLkyeMQPraGAQxj.tmp'
2023-08-05 16:12:10,780 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\.dvc\cache\files\md5\.49odRazVARr5mYNmzgWFgN.tmp'
2023-08-05 16:12:10,795 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file00'
2023-08-05 16:12:11,925 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file01'
2023-08-05 16:12:13,588 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file08'
2023-08-05 16:12:14,388 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file06'
2023-08-05 16:12:15,518 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file02'
2023-08-05 16:12:16,977 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file04'
2023-08-05 16:12:18,451 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file05'
2023-08-05 16:12:19,910 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file09'
2023-08-05 16:12:21,306 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file03'
2023-08-05 16:12:22,797 DEBUG: Removing 'C:\Users\achimgaedke\extCode\dvc_stress_test\data\big_random_file07'
2023-08-05 16:12:24,319 DEBUG: stage: 'create_data' was reproduced
Generating lock file 'dvc.lock'
Updating lock file 'dvc.lock'
To track the changes with git, run:
git add dvc.lock
To enable auto staging, run:
dvc config core.autostage true
Use `dvc push` to send your updates to remote storage.
2023-08-05 16:12:24,335 DEBUG: Analytics is disabled.
In the extension, we can add GIT_OPTIONAL_LOCKS to all of our Git calls. We can also reduce the overall number by introducing a dependency on the Git extension and pulling information from there. The second part will require a few days of work.
The issue in the VS Code repo is https://github.com/iterative/vscode-dvc/issues/4111
After making some updates to the extension (available in 1.0.35) I can run the script from https://github.com/iterative/dvc/issues/9749#issuecomment-1646796239 without error (on a Mac):
https://github.com/iterative/dvc/assets/37993418/18813537-6cfd-4cf0-904f-d73f43ca633e
❯ dvc doctor
DVC version: 3.13.2 (pip)
-------------------------
Platform: Python 3.10.10 on macOS-13.5-arm64-arm-64bit
Subprojects:
dvc_data = 2.12.1
dvc_objects = 0.24.1
dvc_render = 0.5.3
dvc_task = 0.3.0
scmrepo = 1.1.0
Supports:
http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3)
Config:
Global: /Users/mattseddon/Library/Application Support/dvc
System: /Library/Application Support/dvc
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s1s1
Caches: local
Remotes: None
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc, git
Repo.site_cache_dir: /Library/Caches/dvc/repo/ae1da75f7e11ee74c1386b2bc6f713c8
Still the same trouble with DVC 3.14.0 when using VSCode - error message as described last week in https://github.com/iterative/dvc/issues/9749#issuecomment-1666381142
If / When being pedantic about issue management, one could say: The permission error is resolved (close this issue?) and open a new one of cache linking failing under MS Windows/VSCode...
@AchimGaedkeLynker Does that issue still only happen when using VS Code?
Sorry, still happening with 3.15.3
2023-08-21 18:55:53,903 ERROR: failed to reproduce 'create_data': No possible cache link types for 'C:\Users\achimgaedke\extCode\dvc_test\data'. See <https://dvc.org/doc/user-guide/troubleshooting#cache-types> for more information.
Apologies to take that long. And thanks for keeping pushing.
Sorry, the error changes, but still fails when running in VSCode (powershell terminal in VSCode), but not when executing on the standalone powershell terminal (successful logs are attached below).
Is there a way I can make those error reports more productive / repducible for you? e.g. create a screen recording or provide some other setup / windows config information.
2023-09-01 12:05:09,185 DEBUG: v3.17.0, CPython 3.10.11 on Windows-10-10.0.22621-SP0
2023-09-01 12:05:09,205 DEBUG: command: repro -v
2023-09-01 12:05:09,720 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-01 12:05:09,736 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-01 12:05:09,932 DEBUG: Dependency 'create_data_dir.py' of stage: 'create_data' changed because it is 'modified'.
2023-09-01 12:05:09,938 DEBUG: stage: 'create_data' changed.
2023-09-01 12:05:09,946 DEBUG: Removing output 'data' of stage: 'create_data'.
2023-09-01 12:05:09,946 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest_vscode\data'
2023-09-01 12:05:09,954 DEBUG: {'create_data_dir.py': 'modified'}
Running stage 'create_data':
> python create_data_dir.py
2023-09-01 12:05:19,419 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-01 12:05:39,773 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-01 12:05:39,773 DEBUG: Computed stage: 'create_data' md5: '26237bb165f9c74df6920096edc70031'
2023-09-01 12:05:39,841 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-01 12:05:39,848 DEBUG: Preparing to transfer data from 'memory://dvc-staging-md5/f9cae4fd742c4aa89bc9eaca61a72290e723fd0bda6cc4c7f7504540702660a6' to 'C:\Users\achimgaedke\Code\dvc_stresstest_vscode\.dvc\cache\files\md5'
2023-09-01 12:05:39,848 DEBUG: Preparing to collect status from 'C:\Users\achimgaedke\Code\dvc_stresstest_vscode\.dvc\cache\files\md5'
2023-09-01 12:05:39,857 DEBUG: Collecting status from 'C:\Users\achimgaedke\Code\dvc_stresstest_vscode\.dvc\cache\files\md5'
2023-09-01 12:05:39,859 DEBUG: Preparing to collect status from 'memory://dvc-staging-md5/f9cae4fd742c4aa89bc9eaca61a72290e723fd0bda6cc4c7f7504540702660a6'
2023-09-01 12:05:39,865 DEBUG: transfer dir: md5: 625be7c4ce6f1264a616c17b0e648bdc.dir with 10 files
2023-09-01 12:05:39,895 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-01 12:05:39,904 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest_vscode\.UwV6y5bogFkMPtL8DMjVu3.tmp'
2023-09-01 12:05:39,911 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest_vscode\.dvc\cache\files\md5\.VabhQt785HcgHUPzPG6NVh.tmp'
2023-09-01 12:05:39,919 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest_vscode\data\big_random_file09'
2023-09-01 12:05:39,927 ERROR: failed to reproduce 'create_data': [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\achimgaedke\\Code\\dvc_stresstest_vscode\\data\\big_random_file09'
Traceback (most recent call last):
File "dvc_objects\fs\utils.py", line 89, in _unlink
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\achimgaedke\\Code\\dvc_stresstest_vscode\\data\\big_random_file09'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "dvc\repo\reproduce.py", line 199, in _reproduce
File "dvc\repo\reproduce.py", line 129, in _reproduce_stage
File "funcy\decorators.py", line 47, in wrapper
File "dvc\stage\decorators.py", line 43, in rwlocked
File "funcy\decorators.py", line 68, in __call__
File "dvc\stage\__init__.py", line 424, in reproduce
File "funcy\decorators.py", line 47, in wrapper
File "dvc\stage\decorators.py", line 43, in rwlocked
File "funcy\decorators.py", line 68, in __call__
File "dvc\stage\__init__.py", line 617, in run
File "funcy\decorators.py", line 47, in wrapper
File "dvc\stage\decorators.py", line 43, in rwlocked
File "funcy\decorators.py", line 68, in __call__
File "dvc\stage\__init__.py", line 540, in commit
File "dvc\output.py", line 805, in commit
File "dvc\output.py", line 759, in _checkout
File "dvc_data\hashfile\checkout.py", line 270, in checkout
File "dvc_data\hashfile\checkout.py", line 217, in _checkout
File "dvc_data\hashfile\checkout.py", line 88, in _checkout_file
File "dvc_data\hashfile\checkout.py", line 59, in _relink
File "dvc_data\hashfile\checkout.py", line 55, in _remove
File "dvc_objects\fs\base.py", line 476, in rm
File "dvc_objects\fs\local.py", line 121, in rm
File "dvc_objects\fs\utils.py", line 101, in remove
File "dvc_objects\fs\utils.py", line 91, in _unlink
File "dvc_objects\fs\utils.py", line 84, in _chmod
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\achimgaedke\\Code\\dvc_stresstest_vscode\\data\\big_random_file09'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "dvc\cli\__init__.py", line 209, in main
File "dvc\cli\command.py", line 26, in do_run
File "dvc\commands\repro.py", line 13, in run
File "dvc\repo\__init__.py", line 61, in wrapper
File "dvc\repo\scm_context.py", line 151, in run
File "dvc\repo\reproduce.py", line 260, in reproduce
File "dvc\repo\reproduce.py", line 203, in _reproduce
File "dvc\repo\reproduce.py", line 167, in _raise_error
dvc.exceptions.ReproductionError: failed to reproduce 'create_data'
2023-09-01 12:05:39,984 DEBUG: Analytics is disabled.
dvc doctor
DVC version: 3.17.0
-------------------
Platform: Python 3.10.11 on Windows-10-10.0.22621-SP0
Subprojects:
Supports:
azure (adlfs = 2023.8.0, knack = 0.11.0, azure-identity = 1.14.0),
gdrive (pydrive2 = 1.17.0),
gs (gcsfs = 2023.6.0),
hdfs (fsspec = 2023.6.0, pyarrow = 13.0.0),
http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
oss (ossfs = 2023.8.0),
s3 (s3fs = 2023.6.0, boto3 = 1.28.17),
ssh (sshfs = 2023.7.0),
webdav (webdav4 = 0.9.8),
webdavs (webdav4 = 0.9.8),
webhdfs (fsspec = 2023.6.0)
Config:
Global: C:\Users\achimgaedke\AppData\Local\iterative\dvc
System: C:\ProgramData\iterative\dvc
Cache types: hardlink, symlink
Cache directory: NTFS on C:\
Caches: local
Remotes: None
Workspace directory: NTFS on C:\
Repo: dvc, git
Repo.site_cache_dir: C:\ProgramData\iterative\dvc\Cache\repo\b0057e00cc7a7743d344d53d5de1969f
This is the log of the successful run in the windows powershell (executed with the same software versions as the above failed VSCode run):
dvc repro -v
2023-09-01 11:58:11,500 DEBUG: v3.17.0, CPython 3.10.11 on Windows-10-10.0.22621-SP0
2023-09-01 11:58:11,500 DEBUG: command: repro -v
2023-09-01 11:58:11,704 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-01 11:58:11,704 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-01 11:58:11,799 DEBUG: Dependency 'create_data_dir.py' of stage: 'create_data' changed because it is 'modified'.
2023-09-01 11:58:11,799 DEBUG: stage: 'create_data' changed.
2023-09-01 11:58:11,799 DEBUG: Removing output 'data' of stage: 'create_data'.
2023-09-01 11:58:11,799 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data'
2023-09-01 11:58:11,799 DEBUG: {'create_data_dir.py': 'modified'}
Running stage 'create_data':
> python create_data_dir.py
2023-09-01 11:58:18,705 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-01 11:58:45,230 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-01 11:58:45,230 DEBUG: Computed stage: 'create_data' md5: '26237bb165f9c74df6920096edc70031'
2023-09-01 11:58:45,278 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-01 11:58:45,278 DEBUG: Preparing to transfer data from 'memory://dvc-staging-md5/a600f1d29891094036239a26abcb5b30f33f64940a2ad1e321e19f81b5e556ea' to 'C:\Users\achimgaedke\Code\dvc_stresstest\.dvc\cache\files\md5'
2023-09-01 11:58:45,278 DEBUG: Preparing to collect status from 'C:\Users\achimgaedke\Code\dvc_stresstest\.dvc\cache\files\md5'
2023-09-01 11:58:45,278 DEBUG: Collecting status from 'C:\Users\achimgaedke\Code\dvc_stresstest\.dvc\cache\files\md5'
2023-09-01 11:58:45,293 DEBUG: Preparing to collect status from 'memory://dvc-staging-md5/a600f1d29891094036239a26abcb5b30f33f64940a2ad1e321e19f81b5e556ea'
2023-09-01 11:58:45,293 DEBUG: transfer dir: md5: 625be7c4ce6f1264a616c17b0e648bdc.dir with 10 files
2023-09-01 11:58:45,314 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-01 11:58:45,314 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\.F6MBueW2D2BXetwFYaGhKX.tmp'
2023-09-01 11:58:45,314 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\.dvc\cache\files\md5\.6rGPkf75aJ9SNfFwnRqvSr.tmp'
2023-09-01 11:58:45,326 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file01'
2023-09-01 11:58:46,420 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file07'
2023-09-01 11:58:47,497 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file08'
2023-09-01 11:58:48,541 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file04'
2023-09-01 11:58:49,950 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file05'
2023-09-01 11:58:51,432 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file02'
2023-09-01 11:58:52,901 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file03'
2023-09-01 11:58:54,308 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file00'
2023-09-01 11:58:55,831 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file09'
2023-09-01 11:58:57,293 DEBUG: Removing 'C:\Users\achimgaedke\Code\dvc_stresstest\data\big_random_file06'
2023-09-01 11:58:58,818 DEBUG: stage: 'create_data' was reproduced
Generating lock file 'dvc.lock'
Updating lock file 'dvc.lock'
To track the changes with git, run:
git add dvc.lock
To enable auto staging, run:
dvc config core.autostage true
Use `dvc push` to send your updates to remote storage.
2023-09-01 11:58:58,828 DEBUG: Analytics is disabled.
And some diagnostics:
PS C:\Users\achimgaedke\Code\dvc_stresstest> dvc doctor
DVC version: 3.17.0
-------------------
Platform: Python 3.10.11 on Windows-10-10.0.22621-SP0
Subprojects:
Supports:
azure (adlfs = 2023.8.0, knack = 0.11.0, azure-identity = 1.14.0),
gdrive (pydrive2 = 1.17.0),
gs (gcsfs = 2023.6.0),
hdfs (fsspec = 2023.6.0, pyarrow = 13.0.0),
http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
oss (ossfs = 2023.8.0),
s3 (s3fs = 2023.6.0, boto3 = 1.28.17),
ssh (sshfs = 2023.7.0),
webdav (webdav4 = 0.9.8),
webdavs (webdav4 = 0.9.8),
webhdfs (fsspec = 2023.6.0)
Config:
Global: C:\Users\achimgaedke\AppData\Local\iterative\dvc
System: C:\ProgramData\iterative\dvc
Cache types: hardlink, symlink
Cache directory: NTFS on C:\
Caches: local
Remotes: None
Workspace directory: NTFS on C:\
Repo: dvc, git
Repo.site_cache_dir: C:\ProgramData\iterative\dvc\Cache\repo\c8b5cfd28ee3665dc9ba223b1457822b
@AchimGaedkeLynker Thanks for the detailed report. Do you have the DVC extension installed in VS Code? If so, which version? Do you get the error if you uninstall it?
@mattseddon Any other ideas?
Hi there!
The VSCode extension is DVC v1.0.58. Removing the extension avoids the failure.
With the extension installed and using the VSCode terminal: I get this error message from dvc repro -v:
2023-09-30 12:30:57,513 DEBUG: v3.23.0, CPython 3.10.11 on Windows-10-10.0.22621-SP0
2023-09-30 12:30:57,513 DEBUG: command: repro -v
2023-09-30 12:30:57,736 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-30 12:30:57,742 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-30 12:30:57,843 DEBUG: Dependency 'create_data_dir.py' of stage: 'create_data' changed because it is 'modified'.
2023-09-30 12:30:57,843 DEBUG: stage: 'create_data' changed.
2023-09-30 12:30:57,853 DEBUG: Removing output 'data' of stage: 'create_data'.
2023-09-30 12:30:57,853 DEBUG: Removing 'C:\Users\achimgaedke\dvc_stress_test_vscode\data'
2023-09-30 12:30:57,863 DEBUG: {'create_data_dir.py': 'modified'}
Running stage 'create_data':
> python create_data_dir.py
2023-09-30 12:33:10,522 DEBUG: Lockfile for 'dvc.yaml' not found
2023-09-30 12:33:16,442 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-30 12:33:16,442 DEBUG: Computed stage: 'create_data' md5: 'f245734c5a8cc0588b72210046fab8da'
2023-09-30 12:33:16,490 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-30 12:33:16,495 DEBUG: Preparing to transfer data from 'memory://dvc-staging-md5/6f37f50b836981cf8ae7965ff10b89389b3a1dd67f2083d52d7c47882a45c101' to 'C:\Users\achimgaedke\dvc_stress_test_vscode\.dvc\cache\files\md5'
2023-09-30 12:33:16,495 DEBUG: Preparing to collect status from 'C:\Users\achimgaedke\dvc_stress_test_vscode\.dvc\cache\files\md5'
2023-09-30 12:33:16,499 DEBUG: Collecting status from 'C:\Users\achimgaedke\dvc_stress_test_vscode\.dvc\cache\files\md5'
2023-09-30 12:33:16,505 DEBUG: Preparing to collect status from 'memory://dvc-staging-md5/6f37f50b836981cf8ae7965ff10b89389b3a1dd67f2083d52d7c47882a45c101'
2023-09-30 12:33:16,509 DEBUG: transfer dir: md5: 625be7c4ce6f1264a616c17b0e648bdc.dir with 10 files
2023-09-30 12:33:16,544 DEBUG: built tree 'object 625be7c4ce6f1264a616c17b0e648bdc.dir'
2023-09-30 12:33:16,553 DEBUG: Removing 'C:\Users\achimgaedke\dvc_stress_test_vscode\.Lv9wDsT9dbEcJkkoyPcPmt.tmp'
2023-09-30 12:33:16,556 DEBUG: Removing 'C:\Users\achimgaedke\dvc_stress_test_vscode\.dvc\cache\files\md5\.3ZQxAU2E7qFJ2NpWP57iHF.tmp'
2023-09-30 12:33:16,561 DEBUG: Removing 'C:\Users\achimgaedke\dvc_stress_test_vscode\data\big_random_file05'
2023-09-30 12:33:32,173 ERROR: failed to reproduce 'create_data': No possible cache link types for 'C:\Users\achimgaedke\dvc_stress_test_vscode\data'. See <https://dvc.org/doc/user-guide/troubleshooting#cache-types> for more information.
Traceback (most recent call last):
File "dvc\repo\reproduce.py", line 199, in _reproduce
File "dvc\repo\reproduce.py", line 129, in _reproduce_stage
File "funcy\decorators.py", line 47, in wrapper
File "dvc\stage\decorators.py", line 43, in rwlocked
File "funcy\decorators.py", line 68, in __call__
File "dvc\stage\__init__.py", line 436, in reproduce
File "funcy\decorators.py", line 47, in wrapper
File "dvc\stage\decorators.py", line 43, in rwlocked
File "funcy\decorators.py", line 68, in __call__
File "dvc\stage\__init__.py", line 629, in run
File "funcy\decorators.py", line 47, in wrapper
File "dvc\stage\decorators.py", line 43, in rwlocked
File "funcy\decorators.py", line 68, in __call__
File "dvc\stage\__init__.py", line 559, in commit
dvc.exceptions.CacheLinkError: No possible cache link types for 'C:\Users\achimgaedke\dvc_stress_test_vscode\data'. See <https://dvc.org/doc/user-guide/troubleshooting#cache-types> for more information.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "dvc\cli\__init__.py", line 209, in main
File "dvc\cli\command.py", line 26, in do_run
File "dvc\commands\repro.py", line 13, in run
File "dvc\repo\__init__.py", line 61, in wrapper
File "dvc\repo\scm_context.py", line 151, in run
File "dvc\repo\reproduce.py", line 260, in reproduce
File "dvc\repo\reproduce.py", line 203, in _reproduce
File "dvc\repo\reproduce.py", line 167, in _raise_error
dvc.exceptions.ReproductionError: failed to reproduce 'create_data'
2023-09-30 12:33:32,173 DEBUG: Analytics is disabled.
@AchimGaedkeLynker What about with the extension installed but running outside of VS Code?
Just wanted to add I'm experiencing the same issue on Windows 10 with the VSCode Extension, and running outside of VSCode (with it closed) did work. Thanks!
just wondering if there is any update on this. I get the same error but outside VSCode (command console, power console, "as administrator" all with the same result). Only happens when doing an "dvc import" on a Windows PC. "dvc get" works fine (so I assume its again the checkout stage that causes issues). No such problem on linux clients observed.
