Load only hard examples on QuALITY

[yulonglin]

Is there a way to load only the hard examples for QuALITY?

[yulonglin]

Looking through the HuggingFace datasets repo, it seems maybe I could branch off yours and add a "quality_hard" dataset with a QualityConfig that has hard_only=True: https://huggingface.co/datasets/tau/scrolls/blob/main/scrolls.py#L214

But parsing the QuALITY jsonl files I don't see an is_hard attribute/column as is also suggested in your code, so I may be way off the mark.

[UriSha]

Hi,

You are not off the mark at all.

We do evaluate the hard subset of QuALITY (as shown in the leaderboard), but as you noticed we don't allow the user/model to know if an example is from the hard subset or not (our jsonl files don't hold this information).

However, I am pretty sure that you can get this information from the original QuALITY dataset if you want.

I hope that helps!