dvc icon indicating copy to clipboard operation
dvc copied to clipboard

dvc pull with multiple .dvc files into single output folder

Open GLivshits opened this issue 4 months ago • 3 comments

Hello. I have some folder that is already on DVC. It is stored locally like that:

data/release_123/folder_1.dvc

folder_1 has following structure:

data/release_123/folder_1/subfolder_1 data/release_123/folder_1/subfolder_2

So when I run dvc pull data/release_123/folder_1.dvc a folder appears locally: data/release_123/folder_1.

Now I want to add some new folders to folder_1, but also I want to have two separate dvc files for that. For example, lets name it data/release_123/folder_1_new_subfolders.dvc.

I want it work like that:

  1. When I run dvc pull data/release_123/folder_1.dvc, old folder structure appears (like the one above).
  2. When I run dvc pull data/release_123/folder_1_new_subfolders.dvc, newly added folders should be pulled into folder_1.

Final structure that I expect to appear after running 1 and 2 (actually not depending on order):

data/release_123/folder_1/subfolder_1 data/release_123/folder_1/subfolder_2 data/release_123/folder_1/subfolder_3 data/release_123/folder_1/subfolder_4

How to do that? I've tried the following:

  1. Put new subfolders to other folder:

data/release_123/folder_2/subfolder_3 data/release_123/folder_2/subfolder_4

  1. dvc add data/release_123/folder_2
  2. dvc push
  3. Manually renaming "path" field in data/release_123/folder_2.dvc from folder_2 to folder_1.

But in this case, I always get this structure:

data/release_123/folder_1/subfolder_3 data/release_123/folder_1/subfolder_4

when running any of dvc pull data/release_123/folder_1.dvc, dvc pull data/release_123/folder_2.dvc in any order.

GLivshits avatar Sep 30 '24 11:09 GLivshits