list icon indicating copy to clipboard operation
list copied to clipboard

Fetch multiple URLs for a given source

Open attwad opened this issue 4 years ago • 0 comments

Is your feature request related to a problem? Please describe. Some data sets are spread accross multiple files (https://github.com/globaldothealth/list/issues/518 has an example): a file containing case data and a file containing fine grained location data for example.

Describe the solution you'd like Currently the sources can only be configured with one URL that gets retrieved, if we were to make this an array instead we could retrieve multiple documents and pass them all to the parsers which would then join those documents as needed.

That involves schema changes in mongo to store multiple urls, then the retrieval function would need to store and pass multiple s3keys to the parsers. Also involves some UX for how to specify multiple URLs per source.

attwad avatar Sep 17 '20 09:09 attwad