Improve Model

[sam-writer]

Overview

Currently, we use a deterministic SIR model (see sir and sim_sir in models.py) to predict everything. It does not have many parameters, which I think contributes to the ease of use and adoption of the tool... however, accuracy is also of paramount importance. There have been multiple proposed improvements.

Proposed Improvements

• This repo uses MCMC sampling to do more probabilistic models. @sam-qordoba is trying to get the Bayesian SIR model working, but it has obscure requirements. Here is a colab notebook of the main model - the repo works but there are a few setup steps
• Paper suggestion via Google AI: Bayesian Models for Heterogeneous Personalized Health Data - src
• Possibly-useful Transformer model from Google: interpretable multi-horizon forecasting with deep learning, but, " unfortunately for this task at the moment, the amount of data seems very limited and expert human biases (e.g. it takes X days to show symptoms/recover etc.) seem more important." src
• Model should incorporate potential incoming infections from neighboring areas (etc), rather than the assumption of jurisdiction lockdown

Concerns

• "my understanding is the SIR model's more of a guesstimate that can be fit retrospectively but isn't that predictive for changing circumstances. It doesn't account for household contact, or hordes of folks driving their dying relatives from one jam-packed hospital to the next, or the larger consequences of jamming 200 octogenarians into a group home manned by underpaid attendants with a shortage of tests and protective gear. But we're fighting the epidemic blind, so it's what we've got." src
• "There has been a lot of talk about using more complex models, but the hurdles are (1) usability (2) uncertainty/unavailability of the required inputs. I think that the consensus is that better models would be better if they had well-constrianed inputs and didn't make the tool harder for users to adapt to their local contexts. Otherwise better models would be worse." src
• "What are some additional inputs that are missing? Am very interested in modelling with different values for various activities, e.g. school transmission, intrahousehold transmission, workplace transmission. Obviously this is not straightforward. Moreover the SIR model totally leaves out the fact that different populations are more or less vulnerable, if one is looking at hospital capacity there are pretty sizeable regional population variations. For a first order guess it's useful, but the real world is full of cascading impacts that are very hard to guess." src

Definition of Done

This ticket is complete when we have a plan for improving the model, which takes into account the concerns.

[briglass]

Regarding incoming infections, this simulator: http://covidsim.eu/ has an "Infections from outside of the population [per day]", which I am assuming to be accomplished via a monotonic daily increase to S, although I am not sure

[hakocon]

I really like the idea of social distancing (or I was also thinking population density as a proxy based on ZIP or something like that). Also, would it be valuable to bring in the age distribution of a location? This is specifically dangerous for older folks, so locations with a higher number of elderly might see different distributions of acute/critical care patients than those with lower numbers. Just my thoughts, thanks!

[sam-writer]

@hakocon demographics seem very important, I agree, and that is probably just an API call away.

Population distancing as a proxy for social distancing, however, I feel like that assumption could be very wrong - e.g. San Fransisco, with it's shelter in place order, vs somewhere in rural Alaska which, as of today, has a Representative who is saying that fears over "the beer virus" are "blown out of proportion" and that "it’s not nearly as deadly as the other viruses we have."

I don't think we should infer this... but if we were going to automatically infer social distancing for a location, I would rather do it by an algorithm like (pseudocode):

social_distancing_phrases = ["shelter in place", "social distancing", ...]
hoax_phrases = ["hoax", "overblown", ...]

sd_score = 0
hoax_score = 0

def infer_social_distancing(city_name):
    search_results = get_google_top_100_pages_for_term(city_name + " coronavirus")
    texts = get_text_from_pages(search_results)
    for sdp in social_distancing_phrases:
        if sdp in text:
            sd_score += 1
        if hoax_score in text:
            hoax_score += 1
    score = somehow_convert_these_to_a_score(sd_score, hoax_score)
    return score

[selahlynch]

Are daily counts of COVID related hospital/ICU bed usage at Penn Medicine or another hospital system available? If so we can use this to test some of the models.

[cjbayesian]

We need to have an improved way to estimate current # infected. I like the approach taken here: https://github.com/jwrichar/COVID19-mortality/blob/master/State-wise%20Unreported%20COVID-19%20Cases%20in%20the%20U.S..ipynb

[quinn-dougherty]

issue too broad, but has a lot of useful notes and discussion on it so i'm reticent to close it

[sam-writer]

@quinn-dougherty I agree, this is too broad, but the collection of ideas is valuable. Is there some way to make the output of this ticket:

• an artifact, summarizing these ideas and the ideas on slack? Not sure what that would look like, or if this issue is possibly the best format for an ongoing brainstorming session on model improvement...
• a ticket, explaining what model improvements are to be done at this time?