iterative
`dvc queue run <hash>` to run a specific experiment
I'm using the dvc experiment feature together with the queue to plan multiple experiments I want to run. I wanted to run an already queued experiment to check its outputs without submitting all jobs.
My Idea would be something like
dvc queue run <hash> which will run the given experiment in .dvc/tmp/ and upon success remove it from the queue.
Keeping this feature request open, but in case you want a workaround for this in the meantime, you can do:
$ dvc exp apply <queue_id>
$ dvc exp run --temp
Thanks, I had to add dvc queue remove to clean the queue from the old experiment which would just be a copy. But this was more or less exactly what I was looking for.
dvc exp apply $queue_id
dvc exp run --temp
dvc queue remove $queue_id
It probably can't be dvc queue run <hash> because the id changes so I like the dvc queue pop <hash> idea.
How would this affect the workspace? If you use apply + run --temp it will change the workspace, so maybe use stash the workflow before the apply and then apply the stash afterwards?
In addition to dvc queue pop <id> one could also have just dvc queue pop to start the next experiment in line. In this way one can run multiple experiments in temporary directories from the same repository. This could be especially useful on shared file systems to run experiment from different machines.
I found a solution for myself by writing https://github.com/zincware/dask4dvc and adding this feature via dask4dvc run <id> which will reproduce the specific experiment.
Thanks, I had to add
dvc queue removeto clean the queue from the old experiment which would just be a copy. But this was more or less exactly what I was looking for.dvc exp apply $queue_id dvc exp run --temp dvc queue remove $queue_id
does this work in the case of multiple workers assuming the workers aren't trying to do dvc exp apply at the same time? once the call to dvc exp run --temp is called, can the workspace be safely changed by another worker with dvc exp apply?
I'll use tmux for an example:
Assume I already have a list of experiments queued. I open a window, cd to my repo and begin the experiment using dvc exp apply $queue_id, dvc exp run --temp. Can I then open another tmux window and do the same steps again for a different experiment?
Yes, as long as you are okay that each dvc exp apply call will change the state of your workspace. dvc exp run --temp will create a new isolated workspace for each one and run them in parallel.
Okay, is there a command which would restore the workspace to how it was before dvc exp apply was called?
You can git stash --include-untracked before you dvc exp applyand see if git stash pop does work.
@dberenbaum Unfortunately, I won't be able to add my dask4dvc implementation because it doesn't work with celery.
how does dvc deal with this issue? when I queue up experiments with dvc exp run --queue -S param1=val1 ... the workspace is left unchanged afterwards right?
@gregstarr PTAL at https://dvc.org/doc/command-reference/exp/apply#expand-for-details-on-reverting-object-object and see if that helps
yes that does help, thank you!
just to make sure I understand:
-
git stashwould save any changes I made to the experiment - then
git stash apply refs/exps/apply/stashrestores the workspace to how it was prior to thedvc exp apply
is this correct? I'm assuming dvc exp apply saves a stash in "refs/exps/apply/stash" prior to applying the experiment stash? then the second command retrieves it?