dvc
dvc copied to clipboard
Published
20 hours ago •
iterative
iterative
status: repeatedly rehashes files cached as outputs of `foreach` pipeline stages
Bug Report
Description
dvc status repeatedly rehashes files cached as outputs of foreach pipeline stages leading to O(mins) vs O(s) runtimes as compared to dvc data status
Reproduce
run dvc status >=2 times in a repository where the cache is associated to foreach pipeline stages
These are the
where I encountered this issue.
Expected
execution of dvc status completes in O(s) without repeatedly indicating it is computing large file hashes for the same data files that have not changed between executions of status
Environment information
Output of dvc doctor:
output of `dvc doctor`
$ dvc doctor
DVC version: 3.20.0 (conda)
---------------------------
Platform: Python 3.10.12 on Linux-5.15.0-1040-gcp-x86_64-with-glibc2.31
Subprojects:
dvc_data = 2.16.1
dvc_objects = 1.0.1
dvc_render = 0.5.3
dvc_task = 0.3.0
scmrepo = 1.3.1
Supports:
gs (gcsfs = 2023.6.0),
http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3)
Config:
Global: /home/jupyter/.config/dvc
System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sdb
Caches: local
Remotes: gs
Workspace directory: ext4 on /dev/sdb
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/e37b454f284888e332ba9f90041279fd
Additional Information (if any):
time dvc status
> dvc status --verbose
2023-09-15 00:43:05,722 DEBUG: v3.20.0 (conda), CPython 3.10.12 on Linux-5.15.0-1040-gcp-x86_64-with-glibc2.31
2023-09-15 00:43:05,723 DEBUG: command: /opt/conda/envs/project/bin/dvc status --verbose
2023-09-15 00:43:08,520 DEBUG: built tree 'object e20824b47e7c3133fc8feb5afe9768c7.dir'
2023-09-15 00:43:09,007 DEBUG: built tree 'object 9e58457e4977f8efecf2191a9338a944.dir'
2023-09-15 00:43:09,051 DEBUG: built tree 'object a3e920a728c5d0669f3dd2a85b2e1422.dir'
2023-09-15 00:43:09,094 DEBUG: built tree 'object 4105c3f421d85f715413e8af3860b72c.dir'
2023-09-15 00:43:09,160 DEBUG: built tree 'object 6e67f87c909dabd529d34f409396177d.dir'
2023-09-15 00:43:09,204 DEBUG: built tree 'object 02b8e9097f9c90635324f9769f5659a1.dir'
Computing md5 for a large file '/home/user/project/reproducibility/figures/data/processed/d1_processed.h5ad'. This is only done once.
2023-09-15 00:43:12,243 DEBUG: built tree 'object db1ca249c8dc1aec0163fd09dde6be3b.dir'
Computing md5 for a large file '/home/user/project/reproducibility/figures/data/processed/d1_processed.h5ad'. This is only done once.
2023-09-15 00:43:15,289 DEBUG: built tree 'object d1dad7af2c46b8619ce8576c7b94b5a7.dir'
2023-09-15 00:43:15,334 DEBUG: built tree 'object 19e72f681c56d60fb006965fc89ab5e8.dir'
2023-09-15 00:43:15,378 DEBUG: built tree 'object 5164ae52f32a480dc2ce5ba1f046f3b3.dir'
Computing md5 for a large file '/home/user/project/reproducibility/figures/data/processed/d2_processed.h5ad'. This is only done once.
2023-09-15 00:43:24,936 DEBUG: built tree 'object e7ece5ea831b5b9177a93718e599bfb9.dir'
Computing md5 for a large file '/home/user/project/reproducibility/figures/data/processed/d2_processed.h5ad'. This is only done once.
2023-09-15 00:43:34,464 DEBUG: built tree 'object 58c31b611b95498516ebe5e91b5d86c6.dir'
2023-09-15 00:43:34,532 DEBUG: built tree 'object 7834e268f6ef9062b5267fa17d005bb8.dir'
2023-09-15 00:43:34,578 DEBUG: built tree 'object 17ae66e962b4565d2edd372b23612368.dir'
2023-09-15 00:43:34,825 DEBUG: built tree 'object ac34c1d814959e5e5ebd56a4129a5341.dir'
2023-09-15 00:43:35,072 DEBUG: built tree 'object 963f9756bbdd6675802b1b42962a695b.dir'
2023-09-15 00:43:35,284 DEBUG: built tree 'object a106e29aff1384f1fb214ea5eb304569.dir'
2023-09-15 00:43:35,490 DEBUG: built tree 'object f665f411afa30aeaa47e327dfd3a8cb6.dir'
2023-09-15 00:43:36,052 DEBUG: built tree 'object 2f6015ea5a691887dd3ad0be098a87df.dir'
2023-09-15 00:43:36,608 DEBUG: built tree 'object cb70e8ecad5f1bd6694d92597b627f85.dir'
2023-09-15 00:43:36,652 DEBUG: built tree 'object 345e71c61f4537d28b2552cd726aad03.dir'
2023-09-15 00:43:36,718 DEBUG: built tree 'object 3843408f91ad72d018dd891b8fc97df9.dir'
2023-09-15 00:43:37,668 DEBUG: built tree 'object 6aeb7d9d9683f786c319e312300f9352.dir'
2023-09-15 00:43:38,649 DEBUG: built tree 'object 8d86f5995b964e864831c890f45d8f09.dir'
2023-09-15 00:43:39,095 DEBUG: built tree 'object e20824b47e7c3133fc8feb5afe9768c7.dir'
2023-09-15 00:43:39,588 DEBUG: built tree 'object 9e58457e4977f8efecf2191a9338a944.dir'
2023-09-15 00:43:39,632 DEBUG: built tree 'object a3e920a728c5d0669f3dd2a85b2e1422.dir'
2023-09-15 00:43:39,675 DEBUG: built tree 'object 4105c3f421d85f715413e8af3860b72c.dir'
2023-09-15 00:43:39,718 DEBUG: built tree 'object 6e67f87c909dabd529d34f409396177d.dir'
2023-09-15 00:43:39,760 DEBUG: built tree 'object 02b8e9097f9c90635324f9769f5659a1.dir'
Computing md5 for a large file '/home/user/project/reproducibility/figures/data/processed/d1_processed.h5ad'. This is only done once.
2023-09-15 00:43:42,795 DEBUG: built tree 'object db1ca249c8dc1aec0163fd09dde6be3b.dir'
Computing md5 for a large file '/home/user/project/reproducibility/figures/data/processed/d1_processed.h5ad'. This is only done once.
2023-09-15 00:43:45,847 DEBUG: built tree 'object d1dad7af2c46b8619ce8576c7b94b5a7.dir'
2023-09-15 00:43:45,894 DEBUG: built tree 'object 19e72f681c56d60fb006965fc89ab5e8.dir'
2023-09-15 00:43:45,938 DEBUG: built tree 'object 5164ae52f32a480dc2ce5ba1f046f3b3.dir'
Computing md5 for a large file '/home/user/project/reproducibility/figures/data/processed/d2_processed.h5ad'. This is only done once.
2023-09-15 00:43:55,476 DEBUG: built tree 'object e7ece5ea831b5b9177a93718e599bfb9.dir'
Computing md5 for a large file '/home/user/project/reproducibility/figures/data/processed/d2_processed.h5ad'. This is only done once.
2023-09-15 00:44:05,051 DEBUG: built tree 'object 58c31b611b95498516ebe5e91b5d86c6.dir'
2023-09-15 00:44:05,113 DEBUG: built tree 'object 7834e268f6ef9062b5267fa17d005bb8.dir'
2023-09-15 00:44:05,159 DEBUG: built tree 'object 17ae66e962b4565d2edd372b23612368.dir'
2023-09-15 00:44:05,415 DEBUG: built tree 'object ac34c1d814959e5e5ebd56a4129a5341.dir'
2023-09-15 00:44:05,662 DEBUG: built tree 'object 963f9756bbdd6675802b1b42962a695b.dir'
2023-09-15 00:44:05,898 DEBUG: built tree 'object a106e29aff1384f1fb214ea5eb304569.dir'
2023-09-15 00:44:06,111 DEBUG: built tree 'object f665f411afa30aeaa47e327dfd3a8cb6.dir'
2023-09-15 00:44:06,675 DEBUG: built tree 'object 2f6015ea5a691887dd3ad0be098a87df.dir'
2023-09-15 00:44:07,240 DEBUG: built tree 'object cb70e8ecad5f1bd6694d92597b627f85.dir'
2023-09-15 00:44:07,286 DEBUG: built tree 'object 345e71c61f4537d28b2552cd726aad03.dir'
2023-09-15 00:44:07,330 DEBUG: built tree 'object 3843408f91ad72d018dd891b8fc97df9.dir'
2023-09-15 00:44:08,303 DEBUG: built tree 'object 6aeb7d9d9683f786c319e312300f9352.dir'
2023-09-15 00:44:09,277 DEBUG: built tree 'object 8d86f5995b964e864831c890f45d8f09.dir'
2023-09-15 00:44:09,393 DEBUG: built tree 'object 48f2ddc384963e58525508ddbd6c8500.dir'
2023-09-15 00:44:09,452 DEBUG: built tree 'object 6e6f1089260936561cb737a8ece5253c.dir'
2023-09-15 00:44:09,511 DEBUG: built tree 'object 5d50d2f87e62039067ed535c850b8fb6.dir'
2023-09-15 00:44:09,570 DEBUG: built tree 'object d08ea040605c4cb9d12c8b10e63b5f37.dir'
2023-09-15 00:44:09,627 DEBUG: built tree 'object c796ebce26ba26aff89eb9b85e4e5678.dir'
2023-09-15 00:44:09,684 DEBUG: built tree 'object c4c57b79094e06bc3aa4c127ca1fc847.dir'
2023-09-15 00:44:09,742 DEBUG: built tree 'object ae4502a729b109f3c9e7edac7e74e593.dir'
2023-09-15 00:44:09,798 DEBUG: built tree 'object 059259d2c83e5eca65c29e965e0df4ad.dir'
2023-09-15 00:44:09,855 DEBUG: built tree 'object 5cac04b75a2a9d1ba4532a7438877fef.dir'
2023-09-15 00:44:09,934 DEBUG: built tree 'object 46d5354d8d7b767cc54c31af7a297783.dir'
2023-09-15 00:44:09,991 DEBUG: built tree 'object 016e2459c95e6302ac6bd0326123a35c.dir'
2023-09-15 00:44:10,050 DEBUG: built tree 'object 31df6985d31bde9d144055688f96f6ea.dir'
2023-09-15 00:44:10,109 DEBUG: built tree 'object 937c05a633249f50e64fc0181a2bbc51.dir'
2023-09-15 00:44:10,174 DEBUG: built tree 'object 25c907d2804f3cbfb3ce37c4dc481ebe.dir'
2023-09-15 00:44:10,239 DEBUG: built tree 'object 81bac377c3130be2144e7565ae02167b.dir'
2023-09-15 00:44:10,304 DEBUG: built tree 'object 71b82271caeae95d95acd5d22199b188.dir'
2023-09-15 00:44:10,370 DEBUG: built tree 'object f29ccc6b4af0eef9f4ba599f6ff8c58a.dir'
2023-09-15 00:44:10,454 DEBUG: built tree 'object 64429f05b134ce103e54b56cb324a6c2.dir'
2023-09-15 00:44:10,510 DEBUG: built tree 'object c09de38092ec3e1b734d8a2caafa23df.dir'
2023-09-15 00:44:10,569 DEBUG: built tree 'object 4a4a47d931bfc2b2412ab95706b076d6.dir'
Data and pipelines are up to date.
> /usr/bin/time -v dvc status --verbose
2023-09-15 00:45:13,254 DEBUG: v3.20.0 (conda), CPython 3.10.12 on Linux-5.15.0-1040-gcp-x86_64-with-glibc2.31
2023-09-15 00:45:13,254 DEBUG: command: /opt/conda/envs/project/bin/dvc status --verbose
2023-09-15 00:45:15,994 DEBUG: built tree 'object e20824b47e7c3133fc8feb5afe9768c7.dir'
2023-09-15 00:45:16,481 DEBUG: built tree 'object 9e58457e4977f8efecf2191a9338a944.dir'
2023-09-15 00:45:16,523 DEBUG: built tree 'object a3e920a728c5d0669f3dd2a85b2e1422.dir'
2023-09-15 00:45:16,567 DEBUG: built tree 'object 4105c3f421d85f715413e8af3860b72c.dir'
2023-09-15 00:45:16,632 DEBUG: built tree 'object 6e67f87c909dabd529d34f409396177d.dir'
2023-09-15 00:45:16,675 DEBUG: built tree 'object 02b8e9097f9c90635324f9769f5659a1.dir'
Computing md5 for a large file '/home/user/project/reproducibility/figures/data/processed/d1_processed.h5ad'. This is only done once.
2023-09-15 00:45:19,719 DEBUG: built tree 'object db1ca249c8dc1aec0163fd09dde6be3b.dir'
Computing md5 for a large file '/home/user/project/reproducibility/figures/data/processed/d1_processed.h5ad'. This is only done once.
2023-09-15 00:45:22,738 DEBUG: built tree 'object d1dad7af2c46b8619ce8576c7b94b5a7.dir'
2023-09-15 00:45:22,785 DEBUG: built tree 'object 19e72f681c56d60fb006965fc89ab5e8.dir'
2023-09-15 00:45:22,866 DEBUG: built tree 'object 5164ae52f32a480dc2ce5ba1f046f3b3.dir'
Computing md5 for a large file '/home/user/project/reproducibility/figures/data/processed/d2_processed.h5ad'. This is only done once.
2023-09-15 00:45:32,458 DEBUG: built tree 'object e7ece5ea831b5b9177a93718e599bfb9.dir'
Computing md5 for a large file '/home/user/project/reproducibility/figures/data/processed/d2_processed.h5ad'. This is only done once.
2023-09-15 00:45:42,013 DEBUG: built tree 'object 58c31b611b95498516ebe5e91b5d86c6.dir'
2023-09-15 00:45:42,097 DEBUG: built tree 'object 7834e268f6ef9062b5267fa17d005bb8.dir'
2023-09-15 00:45:42,141 DEBUG: built tree 'object 17ae66e962b4565d2edd372b23612368.dir'
2023-09-15 00:45:42,386 DEBUG: built tree 'object ac34c1d814959e5e5ebd56a4129a5341.dir'
2023-09-15 00:45:42,629 DEBUG: built tree 'object 963f9756bbdd6675802b1b42962a695b.dir'
2023-09-15 00:45:42,834 DEBUG: built tree 'object a106e29aff1384f1fb214ea5eb304569.dir'
2023-09-15 00:45:43,058 DEBUG: built tree 'object f665f411afa30aeaa47e327dfd3a8cb6.dir'
2023-09-15 00:45:43,616 DEBUG: built tree 'object 2f6015ea5a691887dd3ad0be098a87df.dir'
2023-09-15 00:45:44,171 DEBUG: built tree 'object cb70e8ecad5f1bd6694d92597b627f85.dir'
2023-09-15 00:45:44,215 DEBUG: built tree 'object 345e71c61f4537d28b2552cd726aad03.dir'
2023-09-15 00:45:44,280 DEBUG: built tree 'object 3843408f91ad72d018dd891b8fc97df9.dir'
2023-09-15 00:45:45,229 DEBUG: built tree 'object 6aeb7d9d9683f786c319e312300f9352.dir'
2023-09-15 00:45:46,175 DEBUG: built tree 'object 8d86f5995b964e864831c890f45d8f09.dir'
2023-09-15 00:45:46,618 DEBUG: built tree 'object e20824b47e7c3133fc8feb5afe9768c7.dir'
2023-09-15 00:45:47,121 DEBUG: built tree 'object 9e58457e4977f8efecf2191a9338a944.dir'
2023-09-15 00:45:47,167 DEBUG: built tree 'object a3e920a728c5d0669f3dd2a85b2e1422.dir'
2023-09-15 00:45:47,211 DEBUG: built tree 'object 4105c3f421d85f715413e8af3860b72c.dir'
2023-09-15 00:45:47,255 DEBUG: built tree 'object 6e67f87c909dabd529d34f409396177d.dir'
2023-09-15 00:45:47,298 DEBUG: built tree 'object 02b8e9097f9c90635324f9769f5659a1.dir'
Computing md5 for a large file '/home/user/project/reproducibility/figures/data/processed/d1_processed.h5ad'. This is only done once.
2023-09-15 00:45:50,307 DEBUG: built tree 'object db1ca249c8dc1aec0163fd09dde6be3b.dir'
Computing md5 for a large file '/home/user/project/reproducibility/figures/data/processed/d1_processed.h5ad'. This is only done once.
2023-09-15 00:45:53,344 DEBUG: built tree 'object d1dad7af2c46b8619ce8576c7b94b5a7.dir'
2023-09-15 00:45:53,389 DEBUG: built tree 'object 19e72f681c56d60fb006965fc89ab5e8.dir'
2023-09-15 00:45:53,433 DEBUG: built tree 'object 5164ae52f32a480dc2ce5ba1f046f3b3.dir'
Computing md5 for a large file '/home/user/project/reproducibility/figures/data/processed/d2_processed.h5ad'. This is only done once.
2023-09-15 00:46:02,955 DEBUG: built tree 'object e7ece5ea831b5b9177a93718e599bfb9.dir'
Computing md5 for a large file '/home/user/project/reproducibility/figures/data/processed/d2_processed.h5ad'. This is only done once.
2023-09-15 00:46:12,560 DEBUG: built tree 'object 58c31b611b95498516ebe5e91b5d86c6.dir'
2023-09-15 00:46:12,605 DEBUG: built tree 'object 7834e268f6ef9062b5267fa17d005bb8.dir'
2023-09-15 00:46:12,647 DEBUG: built tree 'object 17ae66e962b4565d2edd372b23612368.dir'
2023-09-15 00:46:12,917 DEBUG: built tree 'object ac34c1d814959e5e5ebd56a4129a5341.dir'
2023-09-15 00:46:13,165 DEBUG: built tree 'object 963f9756bbdd6675802b1b42962a695b.dir'
2023-09-15 00:46:13,395 DEBUG: built tree 'object a106e29aff1384f1fb214ea5eb304569.dir'
2023-09-15 00:46:13,609 DEBUG: built tree 'object f665f411afa30aeaa47e327dfd3a8cb6.dir'
2023-09-15 00:46:14,184 DEBUG: built tree 'object 2f6015ea5a691887dd3ad0be098a87df.dir'
2023-09-15 00:46:14,761 DEBUG: built tree 'object cb70e8ecad5f1bd6694d92597b627f85.dir'
2023-09-15 00:46:14,824 DEBUG: built tree 'object 345e71c61f4537d28b2552cd726aad03.dir'
2023-09-15 00:46:14,879 DEBUG: built tree 'object 3843408f91ad72d018dd891b8fc97df9.dir'
2023-09-15 00:46:15,821 DEBUG: built tree 'object 6aeb7d9d9683f786c319e312300f9352.dir'
2023-09-15 00:46:16,766 DEBUG: built tree 'object 8d86f5995b964e864831c890f45d8f09.dir'
2023-09-15 00:46:16,884 DEBUG: built tree 'object 48f2ddc384963e58525508ddbd6c8500.dir'
2023-09-15 00:46:16,939 DEBUG: built tree 'object 6e6f1089260936561cb737a8ece5253c.dir'
2023-09-15 00:46:16,995 DEBUG: built tree 'object 5d50d2f87e62039067ed535c850b8fb6.dir'
2023-09-15 00:46:17,057 DEBUG: built tree 'object d08ea040605c4cb9d12c8b10e63b5f37.dir'
2023-09-15 00:46:17,116 DEBUG: built tree 'object c796ebce26ba26aff89eb9b85e4e5678.dir'
2023-09-15 00:46:17,174 DEBUG: built tree 'object c4c57b79094e06bc3aa4c127ca1fc847.dir'
2023-09-15 00:46:17,231 DEBUG: built tree 'object ae4502a729b109f3c9e7edac7e74e593.dir'
2023-09-15 00:46:17,290 DEBUG: built tree 'object 059259d2c83e5eca65c29e965e0df4ad.dir'
2023-09-15 00:46:17,370 DEBUG: built tree 'object 5cac04b75a2a9d1ba4532a7438877fef.dir'
2023-09-15 00:46:17,428 DEBUG: built tree 'object 46d5354d8d7b767cc54c31af7a297783.dir'
2023-09-15 00:46:17,485 DEBUG: built tree 'object 016e2459c95e6302ac6bd0326123a35c.dir'
2023-09-15 00:46:17,542 DEBUG: built tree 'object 31df6985d31bde9d144055688f96f6ea.dir'
2023-09-15 00:46:17,600 DEBUG: built tree 'object 937c05a633249f50e64fc0181a2bbc51.dir'
2023-09-15 00:46:17,657 DEBUG: built tree 'object 25c907d2804f3cbfb3ce37c4dc481ebe.dir'
2023-09-15 00:46:17,715 DEBUG: built tree 'object 81bac377c3130be2144e7565ae02167b.dir'
2023-09-15 00:46:17,777 DEBUG: built tree 'object 71b82271caeae95d95acd5d22199b188.dir'
2023-09-15 00:46:17,838 DEBUG: built tree 'object f29ccc6b4af0eef9f4ba599f6ff8c58a.dir'
2023-09-15 00:46:17,919 DEBUG: built tree 'object 64429f05b134ce103e54b56cb324a6c2.dir'
2023-09-15 00:46:17,977 DEBUG: built tree 'object c09de38092ec3e1b734d8a2caafa23df.dir'
2023-09-15 00:46:18,035 DEBUG: built tree 'object 4a4a47d931bfc2b2412ab95706b076d6.dir'
Data and pipelines are up to date.
Command being timed: "dvc status --verbose"
User time (seconds): 60.05
System time (seconds): 4.85
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:05.27
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 59312
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 56
Minor (reclaiming a frame) page faults: 25316
Voluntary context switches: 148
Involuntary context switches: 176
Swaps: 0
File system inputs: 0
File system outputs: 5408
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
time dvc data status
> /usr/bin/time -v dvc data status --verbose
2023-09-15 00:57:14,669 DEBUG: v3.20.0 (conda), CPython 3.10.12 on Linux-5.15.0-1040-gcp-x86_64-with-glibc2.31
2023-09-15 00:57:14,669 DEBUG: command: /opt/conda/envs/project/bin/dvc data status --verbose
No changes.
(there are changes not tracked by dvc, use "git status" to see)
2023-09-15 00:57:18,470 DEBUG: Analytics is disabled.
Command being timed: "dvc data status --verbose"
User time (seconds): 3.90
System time (seconds): 0.11
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:04.04
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 86156
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 19
Minor (reclaiming a frame) page faults: 19488
Voluntary context switches: 22
Involuntary context switches: 12
Swaps: 0
File system inputs: 0
File system outputs: 768
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
