bigscience-workshop
New prompts for Shades FR/EN
This is a test PR for the BigScience eval hackathon. The goals is to use these three new prompts for crowS-pairs to the following end: 1/ test process/pipeline (I think that works) 2/ get feedback on prompts format: are they acceptable? 3/ how will the metric "other" be defined? Here, we essentially need a count of answers, to convert into the proportion of "sentence A" answers.
(Our goal this week is to get the bias eval done on the revised/upgraded crowS-pairs_multilingual dataset).
The template with id:493b3361-f28d-4489-87b3-b42ae90e5e3a does not have a target or output. You need to add the output as you did for the reset : ||| {{answer_choices[stereo_antistereo]}}
Thank you for the suggestion. I made the change. However, I am not sure I understand whether this is going to have the desired effect for the metric planned ("other").
@aneveol, answer_choices[stereo_antistereo] wouldn't work as the target, since stereo_antistereo is not an integer label of which choice is the most stereotypical answer, but a string indicating whether it is about a stereotypical choice or an anti-stereotypical choice. I think we should always suggest the sent_more (more stereotypical choice) as the target.
I have fixed this and also added your prompts to my PR.
@oskarvanderwal we have merge conflicts that have to be resolved - also I don't see the dataset listed on promptsource. I think this PR needs to be cleaned up before merging. For some reason the PR did not build.
@jzf2101 Sorry, I don't have access to this repository (yet). My PR for CrowS-Pairs can be found here. This PR is about the bias-shades dataset, but I agree we need to clean some things up.
@aneveol Maybe you could rename this PR to "Adding prompts for French and English bias-shades"? Then also
- [x] remove the CrowS-Pairs prompts,
- [x] add BigScienceEvalWG as user, and
- [x] make sure your repository is up-to-date with the original. I can help out with this if you add me as contributor to this repo :)
@jzf2101 I have cleaned this PR and fixed the prompt answer choices. Let me know if there is anything else required for this PR!
@jzf2101 Are there additional changes that need to be made to this PR or can it be merged?
@jzf2101, I have created a new PR for adding French and English prompts without any blocking merge conflicts here: https://github.com/bigscience-workshop/promptsource/pull/837
This PR can be closed.