PyRIT
PyRIT copied to clipboard
Azure
The Python Risk Identification Tool for generative AI (PyRIT) is an open access automation framework to empower security professionals and machine learning engineers to proactively find risks in their...
#### Is your feature request related to a problem? Please describe. We have support for HarmBench text data as described in #270 but at the time decided to hold off...
## Description Adding dataset of prompts used for testing medical bias in AI systems. Dataset is from huggingface (https://huggingface.co/datasets/katielink/EquityMedQA) ## Tests and Documentation Added new unit tests and tested with...
## Description Improving our docs on memory and adding a diagram for our db schema ## Tests and Documentation NA
## Description Adding a new target class for AWS Bedrock Anthropic Claude models. It will only work for Anthropic Claude models as the request body is specific to those, but...
## Description This change creates a full pipeline for performing ensemble scoring with crescendo. Included are two new scorers: EnsembleScorer which is the driver of this change and allows results...
## Description This PR introduces an evaluation framework for PyRIT Scorers. The goals of this PR are: - Setting up the process of retrieving baseline metrics for Scorers based on...
## Description Adding a new Orchestrator that constructs attack prompts based on real-world examples. This orchestrator performs best for informational harms in that center on a consistent narrative (i.e. via...
#### Is your feature request related to a problem? Please describe. Let's say we have a question like "Who is best suited for ?" followed by descriptions of 4 candidates...
FEAT Adding flag parameter to LookBackScorer ## Description The LookBackScorer now has a flag parameter that allows the user to specify whether they want to exclude prompts that include #...
## Description This change introduces a new **Policy Scorer** module to address a key limitation in current LLM-based evaluation systems: the lack of clear, structured scoring criteria. Existing scorers often...
