Support for Multiple References?

[ethanjperez]

Thank you for making this package - it will really make using ROUGE easier! Does the package currently support evaluating with multiple references? I didn't find an example in the repo, and looking through the code, I didn't find implementation for this feature (though I may have missed something).

[pltrdy]

This feature is not (yet?) implemented.

Contributions on this are welcome!

[damnko]

Hi, that would be an interesting feature! Do you know how to currently deal with multiple references? Is that simply the average ROUGE score between the candidate and each of the single references? Thanks!

[pltrdy]

@damnko Hi damnko, unfortunately I've no plan of implementing it myself in the short term. Again, contributions are welcome.

I never used it myself, but I know the original ROUGE (aka. ROUGE-1.5.5) handles multiple references by either averaging scores accross references (default) or using the best score. It's controlled by the -f flag, see https://github.com/pltrdy/files2rouge/blob/master/files2rouge/RELEASE-1.5.5/README.txt#L122

[damnko]

Thanks a lot @pltrdy for your prompt reply! I will look more into it.

[SecNotice]

Thanks a lot @pltrdy for your prompt reply! I will look more into it.

Good afternoon! Do you already have a prototype? Can I offer you some help?

[pltrdy]

Feel free to open a PR, I would be able to review it :)