Added Bary Score Metric

[kadirnar]

trafficstars

Fixed https://github.com/huggingface/evaluate/issues/127

I wrote the documentation code for the bary score metric. While running test codes in local, f_score metric failed. So I will solve it here.

Code References:

• https://github.com/PierreColombo/nlg_eval_via_simi_measures
• https://huggingface.co/spaces/lvwerra/bary_score

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[kadirnar]

Error Message:

ModuleNotFoundError: No module named 'ot'

There is POT library in requirements.txt file.

evaluate==0.1.0
datasets~=2.0
POT
transformers
torch

How can I resolve this error?

[mathemakitten]

Hi @kadirnar, thanks for contributing to evaluate!

Since this is a community metric, do you mind reflecting these changes on the bary_score repo directly here? It would be great to have documentation dropped in where the TODO in this file, and fill in the README.md here.

You can clone the repo with git clone https://huggingface.co/spaces/lvwerra/bary_score and commit + push like usual, and the changes will show up as a pull request here.

Let me know if you have any questions 🤗

[lvwerra]

Thanks for the contribution @kadirnar and thanks @mathemakitten for stepping in! I made the community metric lvwerra/bary_score to help the original contributor see how it could be integrated (see https://github.com/huggingface/evaluate/issues/127). So I am happy if somebody else takes it over and I can delete it.

Since it's quite a new and experimental metric I agree that it's probably better suited for a community metric - which means you also get credit when people use it :) It should only take two steps:

1. Use the CLI to create a new community metric evaluate-cli create "Bary Score"
2. Add all the files into the newly created repo, commit and push.

Then you should see the metric on your account and can be loaded with evaluate.load("YOUR_USERNAME/bary_score"). What do you think? Let me know if you need any help and thanks again for working on this!

[kadirnar]

Hi @mathemakitten @lvwerra

I thought you wanted to add this metric. https://github.com/huggingface/evaluate/issues/127#issuecomment-1151040078 . I worked on this metric to help this repo. I don't plan to do personal work with this metric. You can turn this PR off if it will not be added to the evaluate library. Thanks for your help.😊

[lvwerra]

Sorry for the misunderstanding!

[martysai]

@PierreColombo are there plans to maintain the metrics space in HF? Looks like that the copy-paste of the source code from your repo is not as convenient as evaluate.load call.

[PierreColombo]

idk. There are 3 metrics some of them are available in torch metrics. But i cannot add them now :'( that is why i ask i soemone was willing to do it :)

[martysai]

@PierreColombo

We are developing the NLG evaluation framework and took your implementation as a part of it. Currently I started making corresponding HF Spaces (community metrics as well).

You can check them out later here: https://huggingface.co/pragmaticslab

If you'll have any preferences or the guidance, feel free to reach me out.