ESPEI
ESPEI copied to clipboard
ENH: Allow multiple datasets directories to be specified in YAML input
Sometimes it is useful to load datasets from different filesystem locations, for example if one folder contains hand-curated data and another contains automatically generated data.
In code, it would be pretty simple to handle this. Instead of
from espei.datasets import load_datasets, recursive_glob
directory = '/path/to/directory/'
load_datasets(recursive_glob(directory))
we could do
from itertools import chain
from espei.datasets import load_datasets, recursive_glob
directories = ['/path/to/directory_1/', '/path/to/directory_2/']
load_datasets(chain(*map(recursive_glob, directories)))
Alternatively:
from espei.datasets import load_datasets, recursive_glob
directories = ['/path/to/directory_1/', '/path/to/directory_2/']
globbed_files = sum(map(recursive_glob, directories), [])
load_datasets(globbed_files)