covid_age
covid_age copied to clipboard
timriffe
Israel death counts collection automation
Hi everyone,
Thought I'd ask for help here. I've finally been working on automating the collection of death counts on Israel's side.
Now, the only way to get complete death counts currently is by manually downloading cumulative counts from the following dashboard (In Hebrew): https://datadashboard.health.gov.il/COVID-19/general
If you go on that website and click the following download button:
body > ngx-app > ngx-pages > nb-layout > div.scrollable-container > div > div > div > div > nb-layout-column > ngx-general > section:nth-child(3) > div > ngx-tile-wrapper > ngx-horizontal-bar-by-age-gender > nb-card > nb-card-header > ngx-select-header-infected-level > div:nth-child(1) > excel-download-button > div > button
(Open Developer Tools and search that selector)
Then you get a csv file that includes a column of current cumulative death counts.
I've been manually downloading files for quite a while, because I couldn't figure out how to automate this process. The dashboard page isn't automation-friendly unfortunately. If you can think of ideas of how to do that, it would be greatly appreciated!
What I have currently: https://github.com/eshom/covid-dashboard-processing
What the scripts I wrote do, they download the data I semi-regularly upload to Israel's Google Drive folder. It then processes it to fit our format, And finally saves the output to /data/deaths.csv, which is pretty much ready to be joined with other measures.
A known issue with this data source, is that counts aren't strictly cumulative. At some periods, for some undocumented reasons, deaths were reduced. This was checked, and the change in counts is visible on the raw file.
Pinging you here, because I believe you wanted to get this going? @timriffe
The dashboard web framework was updated, completely breaking what automation efforts I had.
After some debugging figured out that the data you download is generated during runtime in the function downloadToExcelArray.
So it's a good starting point for investigating how to automate this.
This looks like a job for rselenium, or selenium straight in Python. I'll see if one of our student assistants can take a look at it.
I've linked to this post to see if someone claims it. If not, then you and I should set up a call so you can show me what's going on and maybe we can hack through it somehow.
I've linked to this post to see if someone claims it. If not, then you and I should set up a call so you can show me what's going on and maybe we can hack through it somehow.
Hi Tim and eshom,
I think this issue can be closed. the weekly data from Israel are automated and downloaded for all measures.
Thanks.
