PyRIT icon indicating copy to clipboard operation
PyRIT copied to clipboard

FEAT Add JailbreakV_28k dataset from HF

Open AdrGav941 opened this issue 3 months ago • 0 comments

Description

This PR adds support for the JailbreakV_28k dataset to PyRIT. One notable departure from multimodal dataset fetching present here is that we need a local download of the images via a Google Drive download provided by the owners of the HF dataset. The share link to the zip file is in the function comments and this function does not work without this being downloaded locally due to the number of images missing in HF. Unzipping if the extracted file is not present at the provided path is handled, as of right now we do not use HF at all for image download due to the large number of missing images so the zip directory is a mandatory parameter.

Addresses https://github.com/Azure/PyRIT/issues/1007

Changes Made:

  • Added integration for JailbreakV_28k
  • Normalizes and associates the datasets "policy" column with harm-category
  • Allows for filtering on harm categories (policy values)

Files Added/Modified:

  • pyrit/datasets/fetch_jailbreakv_28k_dataset.py - Main implementation
  • pyrit/datasets/init.py - Added exports for new functions
  • tests/unit/datasets/test_fetch_jailbreakv_28k_dataset.py - Unit tests
  • tests\integration\datasets\test_fetch_datasets.py - Integration tests added

Tests and Documentation

  • PyTest parametrized testing for filtering and choice of text field (dataset has jailbreak and redteaming prompts)
  • Dataset mocking with both text fields and policy mapped to harm_category

AdrGav941 avatar Sep 22 '25 22:09 AdrGav941