food-inspections-evaluation icon indicating copy to clipboard operation
food-inspections-evaluation copied to clipboard

Refactor `eval_model` and integrate evaluation function more deeply with `30_glmnet_model.R`

Open geneorama opened this issue 8 years ago • 0 comments

This is in reference to pull request @cash

When we look at the evaluation we look primarily at the "days difference", which is an attempt to quantify how much faster we would find a critical violation over the test period.

There is a secondary component to the evaluation, which is the basis for many of the graphs shown in the white paper. The second component is a summary of day by day inspection counts, and the number of critical violations found shown side-by-side with the simulated violations.

An example of the second summary is shown below, where N is the inspection count, POS indicates the critical violations found, and SIM indicates the "simulated" results from the experiment. In all cases TOT indicates a running cumulative sum.

          date  N NTOT POS POSTOT POS_SIM POSTOT_SIM
 1: 2014-09-02  1    1   0      0       1          1
 2: 2014-09-03 44   45  10     10      22         23
 3: 2014-09-04 50   95   8     18      24         47
 4: 2014-09-05 46  141  12     30       8         55
 5: 2014-09-08 37  178   6     36       8         63
 6: 2014-09-09 40  218   7     43      12         75
 7: 2014-09-10 33  251   3     46       4         79
 8: 2014-09-11 41  292  10     56      11         90
 9: 2014-09-12 42  334   5     61      11        101
10: 2014-09-15 35  369   6     67       5        106
11: 2014-09-16 51  420   9     76       5        111
12: 2014-09-17 24  444   6     82       5        116
13: 2014-09-18 41  485   5     87       7        123
14: 2014-09-19 45  530   8     95      10        133
15: 2014-09-22 34  564   7    102       4        137
16: 2014-09-23 33  597  10    112       2        139
17: 2014-09-24 29  626   6    118       6        145
18: 2014-09-25 48  674   5    123       7        152
19: 2014-09-26 49  723   8    131      11        163
20: 2014-09-29 55  778   9    140       5        168
21: 2014-09-30 32  810   1    141       8        176
22: 2014-10-01 29  839   4    145       3        179
23: 2014-10-02 39  878   6    151       7        186
24: 2014-10-03 46  924   5    156       7        193
25: 2014-10-06  8  932   1    157       0        193
26: 2014-10-07 35  967   5    162       7        200
27: 2014-10-08 48 1015   6    168       9        209
28: 2014-10-09 40 1055   2    170       5        214
29: 2014-10-10 36 1091   3    173       2        216
30: 2014-10-14 30 1121   6    179       3        219
31: 2014-10-15 31 1152   5    184       4        223
32: 2014-10-16 40 1192   3    187       4        227
33: 2014-10-17 43 1235   4    191       5        232
34: 2014-10-20 28 1263   3    194       3        235
35: 2014-10-21 37 1300   8    202       5        240
36: 2014-10-22 37 1337   7    209       4        244
37: 2014-10-23 45 1382   5    214       3        247
38: 2014-10-24 49 1431  11    225       1        248
39: 2014-10-27 33 1464   4    229       3        251
40: 2014-10-28 51 1515  11    240       2        253
41: 2014-10-29 37 1552   6    246       3        256
42: 2014-10-30 42 1594   5    251       1        257
43: 2014-10-31 43 1637   7    258       1        258

It is possible to calculate the "days saved" (the primary evaluation metric) from the table shown above. However, it will be useful to be able to quickly calculate the "days saved" metric directly, because then it's available if someone wants to use it in an objective function or optimization.

So, the point of this issue is to break out the eavl_model into the first and second components, and remove a lot of unclear and unnecessary code in the 30 model script.

geneorama avatar Mar 18 '16 14:03 geneorama