dvc
dvc copied to clipboard
config: support checkout_jobs
Bug Report
checkout: slow checkouts
Description
Checkout copies all files in parallel, leading to disk saturation, and excessive checkout times. E.g. At this time, lsof
for the dvc process shows 331 files open.
Reproduce
dvc pull
Expected
Parallelization in moderation, respecting the jobs: parameter in .dvc/config, or some similar parameter.
Environment information
Output of dvc doctor
:
$ dvc doctor
DVC version: 3.11.1 (pip)
-------------------------
Platform: Python 3.10.10 on Linux-6.1.0-11-amd64-x86_64-with-glibc2.36
Subprojects:
dvc_data = 2.10.1
dvc_objects = 0.24.1
dvc_render = 0.5.3
dvc_task = 0.3.0
scmrepo = 1.1.0
Supports:
http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
ssh (sshfs = 2023.7.0)
Config:
Global: /home/john/.config/dvc
System: /etc/xdg/dvc
Additional Information (if any):
https://discuss.dvc.org/t/is-jobs-n-ignored-on-local-stores/1768