[Time-Series] Autoformer model

Open elisim opened this issue 2 years ago • 1 comments

What does this PR do?

Adding Time Series Autoformer model https://arxiv.org/abs/2106.13008

Related issue: #21890

@kashif :)

Differences between the vanilla

Introduced Series Decomposition in encoder & decoder --- done, waiting for review
Replaced canonical self-attention with autocorrelation block --- done, waiting for review
Added seasonal and trend inputs for the decoder --- added todo places in the code

Mar 02 '23 03:03 elisim

One small open issue left, is adding the series decomposition to the decoder with the trend input. Will do after the initial review :)

Mar 07 '23 07:03 elisim

The documentation is not available anymore as the PR was closed or merged.

Apr 21 '23 09:04 HuggingFaceDocBuilderDev

some of the TF tests are failing and I believe they are unrelated

May 09 '23 07:05 kashif

PR is green

May 15 '23 15:05 elisim

thank you @amyeroberts will get it fixed!

May 16 '23 12:05 kashif

@amyeroberts, thank you for the comprehensive CR! I sincerely appreciate the effort and time you dedicated to thoroughly assessing this pull request.

Will be fixed!

May 16 '23 12:05 elisim

CR changes I did:

Added layer_norm_eps
Model layers are now take the config, except the AutoformerAttention which I wasn't sure about
Better variables names
Addressed questions I could answer

fix-copies is falling because diffs with the time-series-transformer. It's about to decide if to change time-series-transformer here, or to remove "copied from..."

@kashif @amyeroberts

May 21 '23 09:05 elisim