triage icon indicating copy to clipboard operation
triage copied to clipboard

Get an evaluation per as_of_date

Open nanounanue opened this issue 5 years ago • 2 comments

What we want

If you have several as_of_dates in your test matrices, you should get one evaluation per as_of_date.

Rationale

You are training a model every year, but you are using to predict for the full next year, but generating predictions every month. In this moment triage is just storing one evaluation, presumably sorting by score ignoring the date.

Caveat

Probably are scenarios where you want the current behavior, so the proposal is add it in the future (i.e. in a new issue) evaluation_frequency that is optional and by default equal to test_as_of_date_frequencies.

nanounanue avatar Mar 29 '19 23:03 nanounanue

See also #378 #223 for relevant discussions

ecsalomon avatar Mar 30 '19 00:03 ecsalomon

Multiple as of dates in test matrices was never intended to be used this way. Instead, timechop is built to anticipate using multiple test matrices for this purpose. Putting all of the as of dates in a single matrix is wasteful. Creating one matrix per set of as of dates to be evaluated together would allow triage to reuse test matrices for multiple models.

Also, co-opting multi-date matrices for dual purposes prevents event-based problems from having multiple test windows. Using multiple matrices works for all problem types.

ecsalomon avatar Mar 30 '19 00:03 ecsalomon