Fixing invalid transitions for NER data

[joshinh]

I noticed that a recent commit (https://github.com/UKPLab/arxiv2018-bayesian-ensembles/commit/db8f676a4bc8b3f3a4d893ca7f9ac10145bf9573) for this repository removed the fixing of invalid transitions for NER data and instead uses the original NER annotations. I was unable to reproduce the original results using the fixing of invalid transitions. Do the results in paper (https://www.aclweb.org/anthology/D19-1101/) use the original NER annotations?

[edwinrobots]

Hi Nitish,

Yes, it seems my recent ‘fix’ for invalid data made the results slightly different to the paper, so I have commented it out. I believe that the current commit should reproduce the results in the paper. Please let me know if you have any further difficulties.

On 20 Apr 2020, at 11:34, Nitish Joshi [email protected] wrote:

I noticed that a recent commit (db8f676 https://github.com/UKPLab/arxiv2018-bayesian-ensembles/commit/db8f676a4bc8b3f3a4d893ca7f9ac10145bf9573) for this repository removed the fixing of invalid transitions for NER data and instead uses the original NER annotations. I was unable to reproduce the original results using the fixing of invalid transitions. Do the results in paper (https://www.aclweb.org/anthology/D19-1101/ https://www.aclweb.org/anthology/D19-1101/) use the original NER annotations?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/UKPLab/arxiv2018-bayesian-ensembles/issues/4, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABODSU6OUED64UKPXY3DK43RNQQLLANCNFSM4MMKMN5A.

[joshinh]

Hi Edwin,

Thanks! I am able to reproduce the results now.

I also had one more question regarding val/test splits. If I understand correctly, both the val+test splits are used to train the model and the results are reported on the test set. Is that correct? (https://github.com/UKPLab/arxiv2018-bayesian-ensembles/blob/master/src/data/load_data.py#L482 - I saw this but wanted to clarify once)

Edit: If yes, is there any reason why it was done differently to what Nguyen et al 2017 (the HMM-crowd paper) did? Their final model was only trained on the test set and the val set was used solely for tuning the hyperparameters.

Thanks a lot in advance!

[edwinrobots]

Hi Nitish,

This is correct.

On 30 Apr 2020, at 14:47, Nitish Joshi [email protected] wrote:

Hi Edwin,

Thanks! I am able to reproduce the results now.

I also had one more question regarding val/test splits. If I understand correctly, both the val+test splits are used to train the model and the results are reported on the test set. Is that correct? (https://github.com/UKPLab/arxiv2018-bayesian-ensembles/blob/master/src/data/load_data.py#L482 https://github.com/UKPLab/arxiv2018-bayesian-ensembles/blob/master/src/data/load_data.py#L482 - I saw this but wanted to clarify once)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/UKPLab/arxiv2018-bayesian-ensembles/issues/4#issuecomment-621863634, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABODSU2XBGCYC4RPZ625V7LRPF6QNANCNFSM4MMKMN5A.