awesome-llm-unlearning icon indicating copy to clipboard operation
awesome-llm-unlearning copied to clipboard

Published 20 hours ago • verified publisher icon chrisliu298
→

Metadata

A resource repository for machine unlearning in large language models

Awesome Large Language Model Unlearning

Awesome GitHub stars GitHub forks GitHub issues GitHub Last commit

This repository tracks the latest research on machine unlearning in large language models (LLMs). The goal is to offer a comprehensive list of papers, datasets, and resources relevant to the topic.

[!NOTE] If you believe your paper on LLM unlearning is not included, or if you find a mistake, typo, or information that is not up to date, please open an issue or submit a pull request, and I will be happy to update the list.

Table of Contents

  • Table of Contents
  • Papers
    • Methods
      • 2024
      • 2023
      • 2022
      • 2021
    • Surveys and Position Papers
      • 2024
      • 2023
  • Blog Posts
  • Datasets

Papers

Methods

2024

2023

2022

2021

Surveys and Position Papers

2024

2023

Blog Posts

Datasets

  • RWKU
    • Description: This dataset is designed to test large language models' ability to unlearn real-world knowledge. It contains 200 unlearning targets and 13,131 forget probes, including fill-in-the-blank, question-answer, and adversarial attack probes. The task is to forget famous people from Wikipedia. The evaluation suite assesses knowledge memorization and manipulation through regular and adversarial settings, as well as membership inference attacks. It also evaluates general ability, reasoning, truthfulness, factuality, and fluency. The unlearning task does not assume access to the forget or retain corpora, as the pre-training data of most language models is not easily accessible.
    • Links: arXiv, Hugging Face
  • TOFU
    • Description: A synthetic QA dataset of fictitious authors generated by GPT-4. The datasets comes with three splits of the retain/forget sets, including 99/1, 95/5, and 90/10 (in percentage). The dataset also includes questions about real authors and world facts to evaluate the loss of general knowledge after unlearning.
    • Links: arXiv, Hugging Face
  • WMDP
    • Description: A benchmark for assessing hazardous knowledge in biology, chemistry, and cybersecurity, containing about 4000 multiple-choice questions with similar style to MMLU. It also comes with corpora in the three domains.
    • Links: arXiv, Hugging Face
  • MMLU Subsets
    • Description: A task proposed along with the WMDP dataset. The goal is to unlearn (retain) three categories in the MMLU dataset: economics (econometrics and others), physics (math and others), and law (jurisprudence and others). The task requires high-precision unlearning, because the retain sets are categories closely related to the unlearning categories.
    • Links: arXiv, Hugging Face
  • arXiv, GitHub, and copyrighted books corpus
    • Description: A dataset for evaluating approximate unlearning algorithms for pre-trained LLMs. The dataset contains both forget and retain splits of each category, and comes with both in-distribution and general retain sets. The dataset is deisgned for unlearning directly on pre-trained models, as they are random samples from the pre-training dataset of Yi.
    • Links: arXiv, Hugging Face

← Metadata

189
Stars
7
Forks
Watchers

Owner

verified publisher icon chrisliu298

Metadata

A resource repository for machine unlearning in large language models

Back