dvc
dvc copied to clipboard
iterative
dvc exp run --queue needs absolute path
dvc exp run --queue ... needs absolute path for the file in the command, otherwise queued experiments will fail because they cannot find the file from the command (if relative). while dvc exp run works also with relative path
maybe i missed it but this was nowhere documented imo and tbh it's a bit inconsistent.
Are you referring to file arguments passed to commands in your dvc.yaml stages?
Using exp run --queue or exp run --temp runs your experiment in a temporary directory separate from your main repo workspace. If the files are in your repo and are tracked by git or DVC, relative paths will still work, since those files are all included in the temporary workspace. But as you noted, if your command takes a path to a file that is not tracked at all in your repo, you will need to use the absolute path to that file.
you are right, the file was not tracked. now it works. thx for clearing that up. however, when creating a queue of 3 experiments with varying parameters, they all had the same name. thus resulted in only 1 exp when using "dvc exp show". i thought each one would get a unique name automatically.
@walternat1ve Could you please elaborate it? You dvc.yaml file? And the dvc exp run command.
all good. the problem was that i changed the params for each queued experiment but didnt provide them as a dependency in dvc.yaml, therefore they were not treated as new experiments but rather i got the cached result and thus the same name. now it makes sense to me. thx for coming back to this issue.
what can i do when a script or file is in a submodule which is tracked by git? it doesn't copy the content to tmp unfortunately
submodules arent currently supported (https://github.com/iterative/dvc/issues/7186, https://github.com/iterative/dvc/issues/5855) but you should be able to use -C/--copy-paths to copy the specify files you need into the tempdir
thx for the quick reply, i will try this workaround.
edit: @pmrowla i was not able to apply the workaround. i tried adding the full directory of submodule but it said its already there (as far as i can tell the directory was there but not the content). it's probably not feasible to specify each single file. what do you think is the best way?