Can you proivde a script or an explanation to reproduce scores in your paper?

[dongheechoi]

In the paper (https://arxiv.org/abs/2306.11189), you wrote the scores below. Can you kindly provide a way to reproduce this?

For example, with the model you provided in the repo BioREx PubMedBERT model (Original) and BioREx BioLinkBERT model (Preferred), what score can I get? And how can I get the score?

When I run with BioREx PubMedBERT model (Original) using the code you suggest bash scripts/run_test_pred.sh, I got

Overall 966 652 263 314 0.7125683060109289 0.6749482401656315 0.6932482721956407 in the file locaed in "out_result_file" parameter. I think it would be precision, recall, f1 score, but then I am not sure I can get 79.6 in this case(BioRED+8 datasets in your paper).

If I misunderstood something, please let me know. And again, if you provide a specific parameters to reproduce the scores in the paper (including the baseline approaches like TL(Transfer learning) or MTL(Multi-Task Learning), it would be great help for me as well.

[dongheechoi]

Also, if you can specify parameters for the other dataset score, it would be very helpful. For example, BC5CDR, DDI, DrugProt, and others.

[ptlai]

Hi @dongheechoi ,

The score that you display is a typed score, and is located in 'out_biorex_results.txt' The score that is in our paper is the binary score, and located in 'out_biorex_bin_results.txt'

To evaluate the prediction of our models, you can consider our latest leaderboard (https://codalab.lisn.upsaclay.fr/competitions/16381).

For different set + BioRED exps, I used the same parameters.

[dongheechoi]

I am sorry to ask this later, but then, do you mean this file?

And I cannot find 79.6 here, so you didn't put the result on the leaderboard, right?

Also, I am wondering that some of the data in the BioREx test set is in the BioRED validation set, so I am not sure which dataset I can use.

For example, I can find this sentence in BioRED validation file

but I can find the same sentence in test file in ncbi_relation and biorex folders.

[dongheechoi]

I have used https://ftp.ncbi.nlm.nih.gov/pub/lu/BioREx/datasets.zip for the BioREx dataset.

[ptlai]

Hi @dongheechoi ,

I wanted to clarify a few points regarding BioREx.

• We don't put BioREx predictions on the leaderboard.
• The 79.6 score (binary relation score) you mentioned doesn't account for variant relations, as we explained in our paper.
• For BioREx performance (including variant relations and relation types), please refer to our proceedings paper, available at https://zenodo.org/records/10351131.
• Regarding the BC8 BioRED validation set, it's derived from the original BioRED corpus's test set. So you'll notice the same sentences in both.