PyRIT icon indicating copy to clipboard operation
PyRIT copied to clipboard

BUG Refusal scorer has opposite example

Open romanlutz opened this issue 1 year ago • 1 comments

Describe the bug

https://github.com/Azure/PyRIT/blob/ecb51c1438aeecb74b45340e4d3993bf0cf688a0/pyrit/datasets/score/refusal.yaml#L81

This line should be removed. The refusal scorer examples need to be consistent with our expectations for it to work. This example 7 has two inputs which are of opposite categories.

romanlutz avatar Oct 11 '24 22:10 romanlutz

@romanlutz Happy to look at this if help is still needed!

asaadkhaja99 avatar Nov 07 '25 16:11 asaadkhaja99