Write test cases for events and holidays regularization

[noxan]

A collection of test cases to check regularization functions. Contains some unit tests for base functions and several integration tests to ensure higher regularization values results in lower weights (high regularization -> weights close to zero).

Pull request to add tests for issue #133

[ourownstory]

Looking good!

Do you think we could do with lower epochs (like 5) and lower regularization (like 10)?

[noxan]

Thanks for the feedback. I did give it a try with epoch=5 and regularization=10. The problem with so few epochs is that the holiday weights do not take reliable values. Here is a Jupyter Notebook on Colab to see the results, just toggle the line with the holiday setup with and without regularization and a different number of epochs. Also added a synthetic dataset with y=1 for all regular days and y=1000 for all holidays, which might be handy for the unit tests.

https://colab.research.google.com/drive/1Z2KRiFvnL33RqYKLovhGmOodZydIfZ_y?usp=sharing

[ourownstory]

I did give it a try with epoch=5 and regularization=10. The problem with so few epochs is that the holiday weights do not take reliable values.

What do you suggest?

(If we merge events and holidays modeling, we could possibly just test for events regularization, as it's the same mechanism)

[codecov-commenter]

Codecov Report

Merging #707 (57b9a31) into main (9fe84ca) will increase coverage by 1.28%. The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #707      +/-   ##
==========================================
+ Coverage   85.57%   86.85%   +1.28%     
==========================================
  Files          15       16       +1     
  Lines        4602     4724     +122     
==========================================
+ Hits         3938     4103     +165     
+ Misses        664      621      -43     

Impacted Files	Coverage Δ
neuralprophet/configure.py	89.47% <0.00%> (-0.23%)	:arrow_down:
neuralprophet/custom_loss_metrics.py	100.00% <0.00%> (ø)
neuralprophet/hdays.py	99.72% <0.00%> (+<0.01%)	:arrow_up:
neuralprophet/metrics.py	85.11% <0.00%> (+0.06%)	:arrow_up:
neuralprophet/df_utils.py	94.35% <0.00%> (+0.19%)	:arrow_up:
neuralprophet/forecaster.py	84.85% <0.00%> (+0.56%)	:arrow_up:
neuralprophet/time_dataset.py	94.40% <0.00%> (+1.11%)	:arrow_up:
neuralprophet/time_net.py	89.09% <0.00%> (+1.46%)	:arrow_up:
neuralprophet/utils.py	91.47% <0.00%> (+2.69%)	:arrow_up:
neuralprophet/plot_forecast.py	72.77% <0.00%> (+3.02%)	:arrow_up:
... and 1 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[noxan]

Found a way to make it work with epoch=10 using tuned hyperparameters and by reducing the batch size for events with the new artificial dataset.

Removing the second set of tests once #712 is done sounds like a great idea, meanwhile, I recommend keeping both.

[noxan]

Add some events and holidays which do not have a meaningful relationship, rather turn them to noise and assume regularization should reduce their weights.

[noxan]

@ourownstory PR is ready to be reviewed. I've added holidays and events with regular values (therefore meaningless entries) and extended the test cases to expect low weights for those regular entries, as discussed earlier today.

[noxan]

@ourownstory all comments resolved as previously discussed, hope it's good to go, looking forward to your feedback.

[ourownstory]

@noxan LGTM!

The only open question I have is: What prompted the regularization to be reduced from 10 to 0.02? Maybe we should look into standardizing the scale of the regularization parameter to be more consistent.

[noxan]

Had to scale regularization down to make sure the meaningful events keep their weights, seems like the regularization scale scales with the number of events and holidays, I created a follow-up issue, see #723.

Final change: reduced regularization from 0.02 to 0.01 to add more safety for randomness in the tests.