Qrel definitions for multiple fields

[janheinrichmerker]

Is your feature request related to a problem? Please describe. When query document pairs have multiple labels associated with them in their qrels, e.g., relevance and quality, only the relevance labels can be documented with qrels definitions (BaseQrels.qrels_defs()).

Describe the solution you'd like I'd like to document qrels definitions for both relevance and quality, as the labels and descriptions are different.

Describe alternatives you've considered I considered adding separate datasets, one for relevance qrels and one for quality qrels, but that is discouraged according to https://github.com/allenai/ir_datasets/pull/135#issuecomment-976658275.

Additional context none

[seanmacavaney]

What do you suppose is the best way to expose these alternate definitions? Maybe:

dataset.qrels_defs(field="relevance") # default to relevance field, but can provide an alternate field name here too

I've also thought about moving these definitions to the documentation yaml file (but still exposing them via .qrels_defs). Does that sound reasonable to you?

[janheinrichmerker]

I agree it would be nice to have them in the YAML 👍 But having an optional argument in qrels_defs is a nice addition too

[janheinrichmerker]

Closing due to inactivity.