Cleanly drop state data upon failures and continue processing
Fixes https://github.com/microCOVID/microCOVID/issues/1570
This will drop a state-level locale from the dataset if we encounter an error while processing data within it. #1570 was caused by the upstream JHU data no longer sending data from Pitcairn Islands, which is considered a state-level locale under the UK in the JHU dataset.
Deploy Preview for microcov ready!
| Name | Link |
|---|---|
| Latest commit | 6ebaea238e9c932d4534b1f71332ddd1dc23319e |
| Latest deploy log | https://app.netlify.com/sites/microcov/deploys/63309a492c645100083e6b2a |
| Deploy Preview | https://deploy-preview-1571--microcov.netlify.app |
| Preview on mobile | Toggle QR Code...Use your smartphone camera to open QR code link. |
To edit notification comments on pull requests, go to your Netlify site settings.
The proximate cause here is that Pitcairn Islands was dropped from JHU's dataset starting 2022-09-14, and update_prevalence.py assumes we will have data for every region it discovers for every day during the 16 days of history we gather:
$ grep Pitcairn 09-13-2022.csv
,,Pitcairn Islands,United Kingdom,2022-09-14 04:20:44,-24.3768,-128.3242,4,0,,,"Pitcairn Islands, United Kingdom",0.0,0.0
$ grep Pitcairn 09-14-2022.csv
$
(you can check these at https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/09-14-2022.csv and so on)
Tested by:
Before fix - running ./update_prevalence.py locally
- verified script failed with error from #1570
After fix - running ./update_prevalence.py locally
- verified that script completed
- verified that UK loads up in site rendered locally
- verified that stale data warning goes away
- verified that Pitcairn Islands is no longer a choice in UK's 'Region' selection