ITO icon indicating copy to clipboard operation
ITO copied to clipboard

Some benchmarks imported from PWC are mapped to tasks that contain multiple concepts

Open kathrinblagec opened this issue 4 years ago • 0 comments

There are at least two cases were this applies:

  • [ ] Process "Question answering" encompasses both extractive (information retrieval) and abstractive question-answering (language generation) benchmarks
  • [ ] Process "Summarisation" encompasses both extractive (information retrieval) and abstractive summarisaton (language generation) benchmarks

Potential solutions:

  • Divide into subclassses and map benchmarks accordingly (e.g., for question answering: 'extractive question answering' and 'abstractive question' answering)
  • Add question answering as a separate NLP category next to the existing hierarchy (natural language generation, natural language anaylsis...)

kathrinblagec avatar Sep 28 '21 09:09 kathrinblagec