Anti-nan protection in MASS Loss is masking all-nan forecast tensors

[alejandroxag]

There is a difference in the anti-nan protection for the MASE Loss function used by ElementAI and the one we are using:

ElementAI only uses the divide_no_nan function over the scale factor (https://github.com/ElementAI/N-BEATS/blob/04f56c4ca4c144071b94089f7195b1dd606072b0/common/torch/losses.py#L61):

masep = t.mean(t.abs(insample[:, freq:] - insample[:, :-freq]), dim=1) masked_masep_inv = divide_no_nan(mask, masep[:, None]) <--- Anti-nan protection only used over the scale factor return t.mean(t.abs(target - forecast) * masked_masep_inv) <--- No anti-nan protection for the forecast

Our MASELoss function is hiding the nans from the forecast (https://github.com/Nixtla/nixtlats/blob/a3c7442a4c16c255685e158c9347d045f87ffa3b/nixtlats/losses/pytorch.py#L160):

delta_y = t.abs(y - y_hat) scale = t.mean(t.abs(y_insample[:, seasonality:] -
y_insample[:, :-seasonality]), axis=1) mase = divide_no_nan(delta_y, scale[:, None]) <--- Anti-nan protection masks nans coming from the scale and the forecast mase = mase * mask mase = t.mean(mase)

This difference causes a silent bug by setting the loss to zero during training/validation when the forecasts are meaningless.

[kdgutier]

Have you tried adding nan protection to the loss function during training? https://github.com/ElementAI/N-BEATS/blob/master/experiments/trainer.py#62