nixtla
nixtla copied to clipboard
fix:Moirai inference in foundation-time-series-arena
Thanks for evaluating Moirai. There are some issues with the way inference is being done for Moirai:
- Using the mean of sample forecasts instead of median. In GluonTS, metrics involving an absolute error typically use the median of sample forecasts rather than the mean.
- Incorrect default parameters were being used. Our paper uses context_length 1000 and patch size 32 as default hyperparameters for the Monash experiments.
- Extremely small batch size is being used, slowing down the model significantly. Differing batch sizes are being used across the various models - with the new default parameters, a larger batch size should be possible.
This PR fixes these issues, and below contain the updated results. The results for the fix was run on an A100-40G GPU, but the appropriate batch size can be updated when running on your hardware.
| Accuracy | Inference Time | |||||||
|---|---|---|---|---|---|---|---|---|
| Monthly | Weekly | Daily | Hourly | Monthly | Weekly | Daily | Hourly | |
| LagLLama | 1.176 | 0.835 | 0.804 | 0.783 | 9.477 | 0.954 | 2.806 | 6.596 |
| Chronos | 0.960 | 0.709 | 0.652 | 0.735 | 38.581 | 5.081 | 7.908 | 11.662 |
| TimesFM | 0.918 | 0.718 | 0.658 | 0.672 | 0.380 | 0.353 | 0.471 | 0.929 |
| TimeGPT | 0.904 | 0.686 | 0.669 | 0.682 | 0.082 | 0.172 | 0.244 | 0.844 |
| TimeGPT-Long | 0.907 | 0.715 | 0.673 | 0.643 | 0.084 | 0.169 | 0.397 | 1.184 |
| Moirai | 1.221 | 0.884 | 0.792 | 0.798 | 3.115 | 3.103 | 1.790 | 1.601 |
| Moirai (Fix) | 0.973 | 0.750 | 0.676 | 0.723 | 0.606 | 0.643 | 0.499 | 0.866 |
We note a significant improvement in accuracy across all metrics, as well as inference time.
Hi @gorold, thank you for taking the time to run the benchmarks and improve Morai. It's great to see the increase in speed and performance. You probably saw that the Chronos also opened a PR suggesting some changes. We will take a close look at both PR's and answer soon. Meanwhile I included a note on the readme to point readers to this PR.
hey @gorold! thank you. could you please sign the CLA?
Hi @AzulGarza and @mergenthaler ,
Thanks for taking the time to look into this PR. May we ask if there are any updates? Have you verified the performance as we provided? Are we good to merge?