Reproducibility of Experiments in SL-Animals

[NakabayashiTakuya]

Hello. I am attempting to replicate the experiments outlined in the SL-Animals 3-Sets dataset's paper.

After downloading the dataset and preprocessing it with the dataset_scripts/sl_animals.py, I conducted experiments using the provided all_params.json. I encountered the error message "FileNotFoundError: [Errno 2] No such file or directory: './datasets/SL_animal_splits/dataset_3sets_12000/train/'" when using the default settings, prompting me to modify line 392 of data_generation.py to "self.data_folder = './datasets/SL_animal_splits/dataset_3sets_2000/'". I haven't made any changes to the code other than that.

Upon running the experiments multiple times with this configuration, the accuracy reached only around 80%. However, the paper reports an accuracy of 87.45%. Could this discrepancy be attributed to the randomness in the initialization of the model weights? If you have any insights, please share them.

I look forward to hearing from you.