hydra
hydra copied to clipboard
Download datasets/artifacts from a predefined S3 path prior to job run
RIght now users have to explicitly write code to download datasets/artifacts which are needed in their experiments. Implement a functionality to allow users to pass a path to the datasets/artifacts as part hydra configs. This will pre-download everything within the path into the container and expose them under data/