Model not working?

[darsh10]

I see a dev/test slot error of 100.0. The train loss goes down. What exactly is going on?

[andy194673]

The model is working. Please refer to the updated training log and testing result. Please note that if the initial learning rate is small (like 0.001), the slot error rate reduces slow. Change the initial learning rate to 0.005, you will see the model is working.

[darsh10]

Hi, thank you so very much for the response and for the bug fix. Yes, I did try a learning rate of 0.005. Yet I get a consistently 100% slot error rate. Can you share the pytorch version that you used, and the exact command used to create the log files ? The reason being, I get a lot of warnings for the code, and that could be feeding in some silent errors. Also the batch-size used by u is different from the one specified in the default command. So, was hoping to get access to your command.

[andy194673]

Hi, the command I used is the same as the one in readme file. The batch size is 200 or 256 doesn't make huge difference. The pytorch version for this code is 0.3.1. What warning did you get?

[darsh10]

Hi, thank you so much for the reply. I tried pytorch 0.3.1. However, the problem still persists. I don't get the warnings anymore. Yet the dev performance is 100% slot error rate.

[miroblog]

same issue, train/validation loss goes down yet, slot error rate is 100% . the model is fine ~ but something is wrong with get_slot_error function.

[youngornever]

I have the same issue

[jin8]

After changing the score function code in run_woz3.py from slot_tok = '@' + _das[0][:3] + '-' + _das[1] + '-' + _das[2] to slot_tok = 'slot-'+_das.split(':')[1].lower() , it solved the the problem of having slot error:100.000.