KG2E crashes after around 400 epochs

Open ArkDu opened this issue 5 years ago • 2 comments

I run train.py with python train.py -mn kg2e -exp true -device cuda. It first seemed alright, but after around 400 epochs it's mini-test results became NaN or 0.000000. Screen Shot 2020-09-13 at 8 46 50 PM This is kg2e_Test_results.csv after training finished. It indicates that the model stopped working at around 400 epochs

Sep 18 '20 22:09 ArkDu

Hi, @ArkDu, I noticed the default learning rate for KG2E is 0.01, which may have been too high and resulted in a large gradient during the training. Would you like to try decreasing it and see if that helps?

Sep 24 '20 08:09 baxtree

Sure I'll try a smaller learning rate and see how it goes.

Oct 06 '20 22:10 ArkDu