covidex icon indicating copy to clipboard operation
covidex copied to clipboard

Evaluate highlighter on SQuAD and BioASQ

Open rodrigonogueira4 opened this issue 4 years ago • 1 comments

The highlighting service is very similar to a Q&A system: given a question and a document, the model outputs a sentence span that might contain the answer.

Hence, we should evaluate different highlighter models (BioBERT, T5, sciT5) on Q&A datasets such as SQuAD and BioASQ.

rodrigonogueira4 avatar Apr 07 '20 14:04 rodrigonogueira4

since it's unsupervised, we can't expect exact match to be high. we should probably use a metric that measures whether the answer span is included in a highlighted sentence.

kyunghyuncho avatar Apr 07 '20 15:04 kyunghyuncho