FEAT Adding retry functionality to PromptSendingOrchestrator
Description
This PR adds the option in PromptSendingOrchestrator to re-send prompts based on the result of a scorer. For now, this is implemented using a SelfAskRefusalScorer, but in the future we may want to replace this with a generic Scorer and implement more general retry logic. We expose this functionality by adding one new arguments to the send_normalizer_requests_async method: max_retries. The example below shows how to retry until all responses are scored as False (i.e., non-refusal), up to a maximum of 5 retries.
send_prompts_async(
prompt_list=prompt_list,
max_retries=5,
)
Thanks @rlundeen2 for the design and @romanlutz for the help with implementation!
Tests and Documentation
This change is documented in the GCG suffix demo in 1_auxiliary_attacks.ipynb but I have not written tests for it yet.