nixtla icon indicating copy to clipboard operation
nixtla copied to clipboard

fix:Moirai inference in foundation-time-series-arena

Open gorold opened this issue 1 year ago • 4 comments

Thanks for evaluating Moirai. There are some issues with the way inference is being done for Moirai:

  1. Using the mean of sample forecasts instead of median. In GluonTS, metrics involving an absolute error typically use the median of sample forecasts rather than the mean.
  2. Incorrect default parameters were being used. Our paper uses context_length 1000 and patch size 32 as default hyperparameters for the Monash experiments.
  3. Extremely small batch size is being used, slowing down the model significantly. Differing batch sizes are being used across the various models - with the new default parameters, a larger batch size should be possible.

This PR fixes these issues, and below contain the updated results. The results for the fix was run on an A100-40G GPU, but the appropriate batch size can be updated when running on your hardware.

Accuracy Inference Time
Monthly Weekly Daily Hourly Monthly Weekly Daily Hourly
LagLLama 1.176 0.835 0.804 0.783 9.477 0.954 2.806 6.596
Chronos 0.960 0.709 0.652 0.735 38.581 5.081 7.908 11.662
TimesFM 0.918 0.718 0.658 0.672 0.380 0.353 0.471 0.929
TimeGPT 0.904 0.686 0.669 0.682 0.082 0.172 0.244 0.844
TimeGPT-Long 0.907 0.715 0.673 0.643 0.084 0.169 0.397 1.184
Moirai 1.221 0.884 0.792 0.798 3.115 3.103 1.790 1.601
Moirai (Fix) 0.973 0.750 0.676 0.723 0.606 0.643 0.499 0.866

We note a significant improvement in accuracy across all metrics, as well as inference time.

gorold avatar Jun 10 '24 15:06 gorold

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Jun 10 '24 15:06 CLAassistant

Hi @gorold, thank you for taking the time to run the benchmarks and improve Morai. It's great to see the increase in speed and performance. You probably saw that the Chronos also opened a PR suggesting some changes. We will take a close look at both PR's and answer soon. Meanwhile I included a note on the readme to point readers to this PR.

mergenthaler avatar Jun 13 '24 19:06 mergenthaler

hey @gorold! thank you. could you please sign the CLA?

AzulGarza avatar Jul 04 '24 14:07 AzulGarza

Hi @AzulGarza and @mergenthaler ,

Thanks for taking the time to look into this PR. May we ask if there are any updates? Have you verified the performance as we provided? Are we good to merge?

liu-jc avatar Aug 28 '24 09:08 liu-jc