iterative
Wait for lock to free
I'm currently integrating DVC with Bazel, and have an issue with the lockfile.
Namely, if I try and run multiple dvc commands in parallel (e.g., multiple pulls), I crash with:
ERROR: failed to pull data from the cloud - Unable to acquire lock. Most likely another DVC process is running or was terminated abruptly.
Now, the lockfile in-and-of-itself isn't a bad idea, but I'd like a canonical way to express that my dvc commands should wait to acquire the lock, rather than failing.
@Wheest, what's the reason to use multiple dvc pull? Could you please share more information about how you are using bazel?
@Wheest, what's the reason to use multiple
dvc pull? Could you please share more information about how you are using bazel?
I have tests or build steps that have those DVC objects as dependencies. Not all of the users of my repo are necessarily going to run those tests or build those artifacts.
So my build process runs DVC pull on any required objects, to fetch them if they're not already available.
Since Bazel can run these build and test steps in parallel, this means that we can end up having multiple instances of my dvc pull rule running at once. This is where the issue lies.