covid-19-open-data

covid-19-open-data copied to clipboard

Automatic sanity check of data, flagging out-of-range or suspiciously large changes

1

Propose a system that on each run of the pipeline would sanity check data for values that are out of a reasonable range, or with a suspiciously large change from...

geening

As data sources wind down (& perhaps shut down?)... "I haven't seen any winding down yet, but yes this will be an issue in the future. In our data, we...

a27cheung

For R, suggest using fread from data.table

4

Seeing how big this file is, I think data.table::fread('https://storage.googleapis.com/covid19-open-data/v2/main.csv', na.strings = c("", "NA", "#N/A")) may be more appropriate than read.csv for the R-documentation!

demetrios1

UK Information is incorrect

13

19/02/2021 | United Kingdom | calculated total | 4,105,399 (+1,447) | no data | no data | 120,363 (+98) -- | -- | -- | -- | -- | --...

jamesrswift

Variants

1

Hi, Is there currently data across variants and if not, are there plans to integrate it? Thanks!

jzamalloa1

"{"logname": "VenezuelaHumDataSource", "timestamp": "2020-11-18T11:53:07.1124", "classname": "VenezuelaHumDataSource", "loglevel": "error", "source_name": "VenezuelaHumDataSource", "config": {"name": "pipelines.epidemiology.ve_humdata.VenezuelaHumDataSource", "fetch": [{"url": "https://docs.google.com/spreadsheets/d/e/2PACX-1vQI4s0no2TS1dYxbv82nhKD7iz8fbDGwdsOI4kzJ0cg3gjOR51KIw_rNOff97Xic_fRQD41xmsDGUfM/pub?gid=1029482781&single=true&output=csv", "opts": {"ext": "csv"}}], "test": {"location_key_match": "VE_.*"}, "automation": {"job_group": "all", "deferred": true}}, "traceback": "Traceback (most...

themonk911

"{"logname": "AlaskaDataSource", "timestamp": "2020-11-18T11:00:53.2333", "classname": "AlaskaDataSource", "loglevel": "error", "source_name": "AlaskaDataSource", "config": {"name": "pipelines.epidemiology.us_ak_authority.AlaskaDataSource", "fetch": [{"url": "https://www.arcgis.com/sharing/rest/content/items/867f802ce1624b46b40d2bd281490078/data", "opts": {"ext": "xlsx"}}], "parse": {"key": "US_AK", "sheet_name": "Table 1"}, "test": {"skip": true, "location_key_match": "^US_AK$"},...

themonk911

"{"logname": "epidemiology", "timestamp": "2020-11-18T11:09:36.5360", "classname": "DataPipeline", "loglevel": "error", "source_name": "CongoDRCHumdataDataSource", "source_config": {"name": "pipelines.epidemiology.cd_humdata.CongoDRCHumdataDataSource", "fetch": [{"url": "https://docs.google.com/spreadsheets/d/e/2PACX-1vQGOGhcfVhfuE_wP5Y78ZoKd6xPlc0BVt6uEPTNJ_6J_dqI29LlwC2KHFqw-9viNsBN60z7wC5ruOtt/pub?gid=0&single=true&output=csv", "opts": {"ext": "csv"}}], "test": {"location_key_match": "CD_.+"}, "automation": {"job_group": "all", "deferred": true}}, "message": "No output...

themonk911

"{"logname": "epidemiology", "timestamp": "2020-11-18T11:09:41.4108", "classname": "DataPipeline", "loglevel": "error", "source_name": "IraqHumdataDataSource", "source_config": {"name": "pipelines.epidemiology.iq_humdata.IraqHumdataDataSource", "fetch": [{"url": "https://docs.google.com/spreadsheets/d/e/2PACX-1vQh_BwL222rdcpIH2rLPIbvdKLJu3fevAy2L82FHUcl-84w6byWRITQicetYzpqX707EUc3qgAJm7Hr/pub?gid=0&single=true&output=csv", "opts": {"ext": "csv"}}], "test": {"location_key_match": "IQ_.*"}, "automation": {"job_group": "all", "deferred": true}}, "message": "No output...

themonk911

MZ data source failing

1

``` "MozambiqueHumdataDataSource", "source_name": "MozambiqueHumdataDataSource", "config": {"name": "pipelines.epidemiology.mz_humdata.MozambiqueHumdataDataSource", "fetch": [{"url": "https://data.humdata.org/dataset/96f9bc99-28ee-4046-8a96-9f7c1a1462d7/resource/2857979e-a528-429e-b7ce-e4b1c3317718/download/mozambique-covid-19-cases.xlsx", "opts": {"ext": "xlsx"}, "parse": {"skiprows": 1}}], "test": {"skip": true, "metadata_query": "key.str.match(\"MZ_.*\")"}, "automation": {"job_group": "all", "deferred": true}}, "traceback": "Traceback (most recent...

themonk911