Retrospective marginal evaluation of closed loop AID systems using real-world data

[scottleibrand]

I just wrote up an idea for "Retrospective marginal evaluation of closed loop AID systems using real-world data" at https://docs.google.com/document/d/1DbPAP5zNZ0BhPi2OK4NgMeBXdAVBy4-N1uz98rUMuio/edit?usp=sharing

If anyone has any specific suggestions for improvement of the doc, it has comments enabled. Or, if you'd like to discuss the idea more generally, this issue, and/or https://gitter.im/openaps/oref0, would probably be good places for that.

[jimrandomh]

Rather than trying to match counterfactual boluses to actual boluses, it might be simpler to think of this as a simulation, with the positive-deviation part coming from the data. Reframing what you wrote in that document into those terms, the algorithm in that document, as I understand it, is:

• Start with a glucose time-series g:t->mg/dL, an insulin-delivery series I:t->units, and an insulin-effect model Ieffect:(t->units, t)->mg/dL/tick
• Take the positive deviation from model, dev(t)=(g(t)-g(t-1))-Ieffect(I,t)
• Generate an alternate set of insulin deliveries I'(t) and an alternate glucose timeseries g'(t)=g'(t-1)+dev(t)+(I'(t)-I(t)) by simulating the new algorithm's decisions on the new glucose timeseries

One limitation to doing it this way is that the timeseries has to be truncated somewhere (ie after DIA), and the state at the truncation point might be better or worse and is significantly under-weighted. One way to validate that this doesn't introduce a bias would be to train a value function on (BG,IOB,COB) and verify that the new algorithm isn't significantly worse at the truncation points.

[scottleibrand]

That sounds a lot like what I suggested at the end: “calculating an array of all historical deviations from predicted BG impact of the actual historical insulin delivered, and then applying those deviations (which in most cases will predominantly reflect the effect of carbs) to a feed-forward simulation model”. Are we talking about the same thing there?

[jimrandomh]

Yes, I think we are.