[Storage] Upload multiple files to the same bucket with different directories

[Michaelvll]

It would be great if sky storage can support uploading multiple files into the same bucket but different folder in the bucket. Currently, a workaround is to write another program to upload the files first, before running the sky yaml.

for l_path in local_paths:
  subprocess.run(f'gsutil -m cp {l_path} gs://bucket-name/run-{run_id}', shell=True)

In yaml:

file_mounts:
  remote_path: gs://bucket-name/run-{run_id}

[romilbhardwaj]

Is it possible for the user to organize their files into a local hierarchy however they want, then simply specify the root of the hierarchy in the source?

mkdir ~/mydir/run-1/
mv {run1_files_path} ~/mydir/run-1/
...

In YAML:

file_mounts:
  remote_path: 
    source: ~/mydir/

[Michaelvll]

Is it possible for the user to organize their files into a local hierarchy however they want, then simply specify the root of the hierarchy in the source?

mkdir ~/mydir/run-1/
mv {run1_files_path} ~/mydir/run-1/
...

In YAML:

file_mounts:
  remote_path: 
    source: ~/mydir/

Yeah, I proposed that to them as well but seems like they do not really want to change the local hierarchy as they will start thousands of spot tasks, each of which will use some of the files. Moving the files into different folders for each task will make the local folder a bit messy.