rcnn-relation-extraction icon indicating copy to clipboard operation
rcnn-relation-extraction copied to clipboard

Published 20 hours ago • verified publisher icon roomylee

Macro-F1 score is lower than anticipated

Open geetickachauhan opened this issue 6 years ago • 7 comments

http://nlpprogress.com/relationship_extraction.html mentions this GitHub as a reimplementation of the following paper: http://www.aclweb.org/anthology/P16-1072 which achieves a macro F1 of 86.3. Upon running the model off the shelf, we achieve a macro F1 of 0.51. Running the eval script shows a macro F1 of 0.48. Is there something that I am missing in running this code? I am running default hyperparameter settings; is there an ideal set of hyperparameter settings?

geetickachauhan avatar Sep 14 '18 23:09 geetickachauhan

@geetickachauhan Hi, sorry for late. I think there are some mistakes. This repository is not reimplementation of that paper. This is a project that applied Lai's rcnn(https://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/view/9745) to the relation extraction task.

@sebastianruder Hi, Ruder. Nice to meet you! I found out that my project was mentioned in your post through @geetickachauhan. It is a great honor for introducing my project through your wonderful post :+1: : http://nlpprogress.com/english/relationship_extraction.html. However, my project is not reimplementation of BRCNN (Cai et al., 2016), unfortunately. I hope you can fix it and improve the completeness of your post. PS. there are some reimplementation projects for relation extraction: CNN (https://github.com/roomylee/cnn-relation-extraction) and Attention-BiLSTM (https://github.com/SeoSangwoo/Attention-Based-BiLSTM-relation-extraction). I would really appreciate it if you introduce them to your post together. :smile:

roomylee avatar Oct 26 '18 06:10 roomylee

Hi Joohong, thanks a lot for the note! Could you add a PR to correct the entry in the table? I think you're better able to describe your repo than me. 😄Also feel free to add the other reimplementation links, as I'm not sure which models they are associated with.

sebastianruder avatar Oct 26 '18 08:10 sebastianruder

I will make a pull request, ASAP! Thanks a lot, Ruder. :smile:

roomylee avatar Oct 26 '18 08:10 roomylee

I add a PR. Please check it. Thanks! @sebastianruder 👍

roomylee avatar Oct 29 '18 18:10 roomylee

Thanks for updating this, everyone! Also @roomylee have you noticed an issue with re-implementing papers that claimed to be achieving macro F1's of 86-88%? The performance for one such re-implementation by my lab mate was in the low 80s.

geetickachauhan avatar Oct 30 '18 00:10 geetickachauhan

@geetickachauhan If hyper-parameter tuning is done well, it may be overcome. L2 regularization may help you, too. It is good to refer to my cnn-relation-extraction and SeoSangwoo's Attention-Based-BiLSTM-relation-extraction.

roomylee avatar Oct 30 '18 06:10 roomylee

Thanks for the PR (https://github.com/sebastianruder/NLP-progress/pull/144), @roomylee! :)

sebastianruder avatar Oct 30 '18 07:10 sebastianruder