PyRIT FEAT: Add SorryBench Dataset

This dataset should be available within PyRIT: https://huggingface.co/datasets/sorry-bench/sorry-bench-202406) Associated paper: https://arxiv.org/pdf/2502.04322v1

[[Content Warning: Prompts are aimed at provoking the model, and may contain offensive content.]] Additional Disclaimer: Given the content of these prompts, keep in mind that you may want to check with your relevant legal department before trying them against LLMs.

Feb 12 '25 17:02 jbolor21

I think I can take care of this by the start of next week

Mar 05 '25 07:03 Jarro01X

Fantastic. Reach out if you have questions

Mar 07 '25 20:03 romanlutz

@Jarro01X any updates on this? No hurry, just want to keep the issues up to date so if you're no longer working on it we can free it up for someone else.

Apr 04 '25 06:04 romanlutz

Hi @romanlutz ! Yes, I was working on it yesterday. I finished it, so tonight I'll run the pre commit hooks/checks and create the PR for this issue and another PR for harm_categories in the babel dataset.

Sorry that this has taken me a while!

Apr 04 '25 17:04 Jarro01X

No worries at all! There's no hurry. I just like to keep issues fresh in case someone is looking for one to pick up. Please take your time!

Apr 05 '25 03:04 romanlutz

@Jarro01X can you let us know if you're still planning to do this? Last time, it sounded like you were pretty close 🙂

Jun 09 '25 23:06 romanlutz

I can take this up if there are no updates otherwise I'll look for some other help-wanted and good-first-issue :)

Aug 11 '25 07:08 drkg4b

Sounds good! Go ahead

Aug 16 '25 21:08 romanlutz

@romanlutz Would like to pick this up since it seems there has been no updates

Nov 02 '25 23:11 0xm00n

Please go ahead 🙂 if you have questions or run into problems do not hesitate to reach out.

Nov 03 '25 00:11 romanlutz

@romanlutz submitted my PR! One update is that the original dataset link is outdated now and the latest is 2025/03 which is what I used in the PR.

Nov 03 '25 18:11 0xm00n