unit8co
[QUESTION] Speed up scoring in ForecastAnomalyModel
Speed up scoring in ForecastAnomalyModel
Custom stride
Is there any reason for having a hard-coded stride (i.e. 1, code here) when running the historical_forecasts in ForecastingAnomalyModel? If the underlying model has a horizon of 10, I might as well set the stride to 10 to have 10x faster scoring processing.
Hi @mcapuccini, yes and no :)
Why Yes: It was the most robust parameter choice with the lowest implementation complexity.
Using a stride=1 and last_points_only=True returns a contiguous target series (a single TimeSeries, not a list of forecast TimeSeries). The values represent to the last predicted values from the historical forecasts.
That means that all values were subject to the same forecast uncertainties. This "consistency" can play a crucial role when fitting the anomaly scorers., since they are trained on the error (or other statics) between the forecasts and the actuals.
If we used a stride=horizon, we must set last_points_only=False. This gives us one contiguous target series, where the values represent all historical forecasts concatenated along the time axis.
This might result in for example having a "ramping" sensation in forecast errors (higher errors the further ahead the prediction) -> non consistent model uncertainty -> can have a negative effect on scorer fitting & prediction.
So, all in all, we chose to fix it for the beginning, to have a robust behavior and avoid some pitfalls (at the cost of efficiency).
Why No: As you say, using stride=horizon can drastically speed up the process, and I assume for many use cases this approach is still valid. We could think of adding support for this. WDYT @madtoinou?
Yeah, let's add it to the roadmap. To make our life maybe a bit easier, I would only support stride=horizon and not stride<horizon to avoid overlapping forecasts and have to arbitrary select one over the other. Then, we would have to extensively check that everything behave as expected, especially at the "junctions" and as you said, that we don't have too much drift along the horizon dimension (which is pretty much the cost of the time gain).
I see your point. With a windowed scorer though one could match the size of the window with the lookback plus the forecasting horizion. As I understand k-means scorer considers the forecasts for each window individually hence the accumulated forecasting error due an horizon >1 should be the same if the window matches lookback plus horizon. Does this make sense?
@dennisbader I am trying out what you suggested but concatenating the historical forecast is also painfully slow:
def concatenate_predictions(
historical_forecasts: List[List[TimeSeries]],
) -> List[TimeSeries]:
# Figure out tot iterations
total_iterations = sum(len(pred_list) - 1 for pred_list in historical_forecasts)
concatenated_forecasts = []
with tqdm(total=total_iterations, desc="Concatenating predictions") as pbar:
for pred_list in historical_forecasts:
concatenated = pred_list[0]
for pred in pred_list[1:]:
concatenated = concatenated.append(pred)
pbar.update(1)
concatenated_forecasts.append(concatenated)
return concatenated_forecasts
Is there anything I can do to run this faster?
If your forecasts are contiguous in time the you can use darts.concatenate on the list of forecasts :)
Cool! I assume that the results from historical_forecasts are contiguous in time. You meant trying something like this right?
historical_forecasts = model.historical_forecasts(
series=train_target_series,
past_covariates=train_past_covariates,
future_covariates=train_future_covariates,
forecast_horizon=10,
stride=10,
last_points_only=False,
retrain=False,
verbose=True,
)
historical_forecasts = [concatenate(ts_list) for ts_list in historical_forecasts]
scorer.fit_from_prediction(train_target_series, historical_forecasts)
Yes, exactly :) Yes, they will be contiguous with this config since forecast_horizon == stride.
(I hope using concatenate sped up things drastically?) 😉