triage
triage copied to clipboard
Get an evaluation per as_of_date
What we want
If you have several as_of_dates
in your test matrices, you should get one evaluation per as_of_date
.
Rationale
You are training a model every year, but you are using to predict for the full next year, but generating predictions every month. In this moment triage
is just storing one evaluation, presumably sorting by score ignoring the date.
Caveat
Probably are scenarios where you want the current behavior, so the proposal is add it in the future (i.e. in a new issue) evaluation_frequency
that is optional and by default equal to test_as_of_date_frequencies
.
See also #378 #223 for relevant discussions
Multiple as of dates in test matrices was never intended to be used this way. Instead, timechop is built to anticipate using multiple test matrices for this purpose. Putting all of the as of dates in a single matrix is wasteful. Creating one matrix per set of as of dates to be evaluated together would allow triage to reuse test matrices for multiple models.
Also, co-opting multi-date matrices for dual purposes prevents event-based problems from having multiple test windows. Using multiple matrices works for all problem types.