dvc icon indicating copy to clipboard operation
dvc copied to clipboard

bench: add repro test

Open dberenbaum opened this issue 1 year ago • 2 comments

Closes https://github.com/iterative/dvc-bench/issues/468

Edit: the need for this is to help track performance for issues like https://github.com/iterative/dvc/issues/9085

dberenbaum avatar Dec 14 '23 19:12 dberenbaum

Codecov Report

Attention: 8 lines in your changes are missing coverage. Please review.

Comparison is base (f3fa674) 90.62% compared to head (d9abbe7) 90.60%. Report is 2 commits behind head on main.

Files Patch % Lines
dvc/testing/benchmarks/cli/commands/test_repro.py 0.00% 8 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10167      +/-   ##
==========================================
- Coverage   90.62%   90.60%   -0.02%     
==========================================
  Files         500      501       +1     
  Lines       37852    37879      +27     
  Branches     5503     5508       +5     
==========================================
+ Hits        34305    34322      +17     
- Misses       2908     2917       +9     
- Partials      639      640       +1     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Dec 14 '23 19:12 codecov[bot]

It's going to be hard to write a fair benchmark for dvc repro.

I understand the motivation, but what you are testing here is dvc commit and that too is a no-op here because all of those were run already with run_copy() and is cached (mtimes, md5 hashes and the actual files).

Even if you made all of them --no-exec and tried to benchmark dvc repro, there is I/O happening in the subprocess cmd, which will be a significant part of the runtime.

skshetry avatar Dec 15 '23 10:12 skshetry