Unable to Reproduce Training Results

[derektan95]

trafficstars

Hi @acl21,

Thanks for sharing this amazing repository! I have tried to train your default code (on both main and dev branches), but was unable to obtain good results. Here is a screenshot of some of the logged training curves.

You may also find other training data and videos here: https://drive.google.com/drive/folders/1xP1IpPDGsw4CwjiPVpauKRN_CL0smQHQ?usp=sharing

Warm Regards, Derek

[acl21]

Hi @derektan95, I am glad you find the repository helpful. 🙂

Firstly, please only use main for all training. Secondly, I am sorry for and surprised by the lack of good reproducibility. I have kept the fundamental aspects of the WCSAC algorithm the same since my last training run in August 2022, so I am slightly confused. As you can see in the commit history, despite a few contributions to the repo since then (thanks to davidferrari95), the core parts remain the same, so this shouldn't happen ideally. 😕

I am currently on vacation until mid-April, and I will re-run the experiments and benchmark the algorithm once again. From my experience, the performance of WCSAC is sensitive to hyperparameters so I assume you used the same default hyperparameters in your experiments. Nonetheless, please add the config files you used for training to the drive you shared and I will take a closer look after my vacation.

Once again, sorry for the trouble. Good luck!

Best regards, Akshay

[derektan95]

Hi @acl21,

Thanks for your reply! I managed to rerun the experiments from the main branch, using the default config file as shown here. However, just as with the dev branch, I was unable to reproduce your results. You may refer to the new training graphs and simulation videos here.

Looking forward to hearing from you on how to make these results reproducible. Thanks!

Warm Regards, Derek