microCOVID icon indicating copy to clipboard operation
microCOVID copied to clipboard

Cleanly drop state data upon failures and continue processing

Open apiology opened this issue 3 years ago • 2 comments

Fixes https://github.com/microCOVID/microCOVID/issues/1570

This will drop a state-level locale from the dataset if we encounter an error while processing data within it. #1570 was caused by the upstream JHU data no longer sending data from Pitcairn Islands, which is considered a state-level locale under the UK in the JHU dataset.

apiology avatar Sep 21 '22 12:09 apiology

Deploy Preview for microcov ready!

Name Link
Latest commit 6ebaea238e9c932d4534b1f71332ddd1dc23319e
Latest deploy log https://app.netlify.com/sites/microcov/deploys/63309a492c645100083e6b2a
Deploy Preview https://deploy-preview-1571--microcov.netlify.app
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify[bot] avatar Sep 21 '22 12:09 netlify[bot]

The proximate cause here is that Pitcairn Islands was dropped from JHU's dataset starting 2022-09-14, and update_prevalence.py assumes we will have data for every region it discovers for every day during the 16 days of history we gather:

$ grep Pitcairn 09-13-2022.csv
,,Pitcairn Islands,United Kingdom,2022-09-14 04:20:44,-24.3768,-128.3242,4,0,,,"Pitcairn Islands, United Kingdom",0.0,0.0
$ grep Pitcairn 09-14-2022.csv
$ 

(you can check these at https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/09-14-2022.csv and so on)

Tested by:

Before fix - running ./update_prevalence.py locally

  • verified script failed with error from #1570

After fix - running ./update_prevalence.py locally

  • verified that script completed
  • verified that UK loads up in site rendered locally
  • verified that stale data warning goes away
  • verified that Pitcairn Islands is no longer a choice in UK's 'Region' selection

apiology avatar Sep 21 '22 12:09 apiology