There seems to be a bug in run.py on eval_output_path

[sythello]

In line 102, eval.main(eval_config) the eval_output_path is replaced (eval.py line 25): args.output.replace('__LOGDIR__', real_logdir)

However, in line 104 res_json = json.load(open(eval_output_path)) the eval_output_path is not replaced.

I got a FileNotFoundError during eval on line 104 and I wonder if this is the reason.

Thanks!

[mellahysf]

Hi,

I'm facing the same problem !! @sythello did you resolve it? If yes how, please?

Thanks

[Sea-Snell]

I just commented that line out, and I moved the print line after it into utils/evalutation.py. Those lines of code are relatively unimportant, they just print the accuracy, but the accuracy is also saved in the output json file so you are not really losing much if you just delete those two lines altogether

[mellahysf]

Ok, @Sea-Snell thank you for this proposition. I did the following thing to keep printing the accuracy : in the file experiments/spider-glove-run.jsonnet, I replaced the value of eval_output by "logdir/glove_run/bs=20,lr=7.4e-04,end_lr=0e0,att/ie_dirs" ant it works.