Unable to reproduce results for spacetimeformer

[TayyabaZainab0807]

I am using the following command that is provided by this repo (only difference is the batch_size)

python train.py spacetimeformer solar_energy --context_points 168 --target_points 24 --d_model 100 --d_ff 400 --enc_layers 5 --dec_layers 5 --l2_coeff 1e-3 --dropout_ff .2 --dropout_emb .1 --d_qk 20 --d_v 20 --n_heads 6 --run_name spatiotemporal_al_solar --batch_size 3 --class_loss_imp 0 --initial_downsample_convs 1 --decay_factor .8 --warmup_steps 1000

I am getting the following results (where as I am expecting MSE: ~7.75): test/acc -1.0 test/class_loss 0.0 test/forecast_loss 0.08704246580600739 test/loss 0.08704246580600739 test/mae 1.7290080221756612 test/mape 21375719.51865129 test/mse 9.604532779042728 test/norm_mae 0.1794128092716004 test/norm_mse 0.0870416207133817 test/recon_loss -1.0 test/smape 1.4066449396255207

[jakegrigsby]

There is a big difference between a batch size of 32 and a batch size of 3!

This is probably the most memory-intensive dataset in the paper. It should be possible to get closer to 7.75 using a larger batch size but a smaller model with more of the memory saving tricks

[TayyabaZainab0807]

Thanks, that works for spacetimeformer model. I am not able to find the right commands to replicate the other results for lstm, lstnet etc. Are you able to provide that as well?