list
list copied to clipboard
Fetch multiple URLs for a given source
Is your feature request related to a problem? Please describe. Some data sets are spread accross multiple files (https://github.com/globaldothealth/list/issues/518 has an example): a file containing case data and a file containing fine grained location data for example.
Describe the solution you'd like Currently the sources can only be configured with one URL that gets retrieved, if we were to make this an array instead we could retrieve multiple documents and pass them all to the parsers which would then join those documents as needed.
That involves schema changes in mongo to store multiple urls, then the retrieval function would need to store and pass multiple s3keys to the parsers. Also involves some UX for how to specify multiple URLs per source.