One day horizon evaluation on hold out week (not backtesting)

[drewnoff]

Checklist

• [X] I have searched the issues.
• [X] I have searched the docs.

❓ Questions and Help

Let's say I have a ts with D frequency and I want my model to forecast one day ahead with all the features available by current day, the day I make forecasting for tomorrow. And I decided to hold out test_ts of last two weeks for evaluating the trained model. And the way I want to evaluate the model is that I suppose that by each day in hold out ts there should be all the features available up until preceding day (like it suppose to be in my production usage). And I don't want to retrain the model after each day (I'm not looking for backtest(..., n_folds=14)).

The sample code I found in the documentation doesn't make what I want, for example it doesn't provide information about one day lags, last seven days mean etc.

HORIZON = 14
model = Pipeline(
                model=model,
                transforms=transforms,
                horizon=HORIZON)
model.fit(train_ts)
forecast_ts = model.forecast()
forecast_ts = model.forecast(future_ts)
...
calc_metric(y_true=test_ts, y_pred=forecast_ts)

It's like I want my pipeline to predict for 1d horizon 14 times to cover test_ts. Is it possible using etna API to achieve what I want?

[martins0n]

Hi! Seems it coudn't be done now.

Most similiar way to do it with existed code base is to use AutoRegressivePipeline - in case of known features and lags with lenght more than horizon it would give the same results as you wish.

We'are planning to add more strategies of backtesting, and we will keep this case in mind too.

[drewnoff]

I think the simplest way to achieve this is using backtest without retraining. I'm not sure whether it's possible right now. Maybe backtest with fine grained different retraining strategies would cover a lot of usage scenarios. For example with this set of parameters: forecasting horizon, retrain period, n_folds

[Mr-Geekman]

Waiting for implementation of inference mode.

[Mr-Geekman]

It is now possible after completion of https://github.com/tinkoff-ai/etna/milestone/23