MulT-MOSEI

[efthymisgeo]

Hello. I have tried to reproduce MulT (both aligned and unaligned) based on your code and I cannot match the (average) results you have reported in the paper. I have tried several seeds but still i am not able to match your performance. I tested the code in an Ubuntu20.04 machine and in particular within a conda environment (python 3.10) with all the corresponding requirements satisfied. Finally, i have cloned the repo (no editing) instead of using the command line tool.

My main concern here is that acc_2 and F1 metrics are almost 1% less than yours. This result is of course statistically significant (the corresponding std's are shown after the comma char). Do you have any good reason why am I getting that much of a difference? Could it be aligned /unaligned, any missing hyperparameters, pytorch version etc (mine is 1.13) ??

Model	Non0_acc_2	Non0_F1_score	Mult_acc_5	Mult_acc_7	MAE	Corr	Data Setting
mult-results.md (yours)	84.63	84.52	54.18	52.84	55.93	73.31	Unaligned
mult (myself)	83.69, 0.25	83.64, 0.21	53.94, 0.42	52.66, 0.37	56.08, 0.32	73.17, 0.51	Unaligned

Thanks in advance.

[py-rgb]

Due to variations in experimental environments, it is normal for experimental results to fluctuate. A decrease of 1.13% is within an acceptable range. MMSA provides config_tune to assist you in selecting the best results specific to your experimental environment.