Question regarding final_beam_acc and BEM

[awasthiabhijeet]

Hi @OhadRubin ,

Could you please let me know whether the final_beam_acc metric in the code is the same as the BEM metric reported in Table 3 of your paper?

I'm finding final_beam_acc it to be much lesser than the EM and BEM scores. Is this expected?

{ "best_epoch": 204, "peak_worker_0_memory_MB": 11110.15234375, "peak_gpu_0_memory_MB": 17076.2431640625, "training_duration": "1 day, 11:19:15.241460", "training_start_epoch": 0, "training_epochs": 304, "epoch": 304, "training_final_beam_acc": 0.9714408725602756, "training_loss": 1.5601123754273762, "training_worker_0_memory_MB": 11110.15234375, "training_gpu_0_memory_MB": 17076.2431640625, "validation_final_beam_acc": 0.6634335596508244, "validation_spider": 0.7177497575169738, "validation_reranker": 0.5470417070805044, "validation_leafs_acc": 0.965082444228904, "validation_loss": 0.0, "best_validation_final_beam_acc": 0.6702230843840931, "best_validation_spider": 0.7381183317167799, "best_validation_reranker": 0.5606207565470417, "best_validation_leafs_acc": 0.9515033947623667, "best_validation_loss": 0.0 }