Home

Welcome to the DP2Unlearning wiki!

This repository provides the DP2Unlearning framework, an innovative solution designed to efficiently and reliably "unlearn" unwanted data from Large Language Models (LLMs) while maintaining model performance and ensuring privacy guarantees. The primary objective of DP2Unlearning is to address ethical and legal challenges associated with data privacy, particularly in cases where a model needs to forget specific data, such as personal information or copyrighted material.

What is DP2Unlearning? DP2Unlearning is a formal framework for unlearning in LLMs using ϵ-differential privacy (DP). It is designed to guarantee effective forgetting of specific data while maintaining the overall utility of the model. Unlike traditional methods, DP2Unlearning offers formal forgetting guarantees with significantly reduced computational costs compared to retraining models from scratch.

The framework uses ϵ-DP protection during the LLM's initial training, which allows for efficient unlearning without retraining. This method provides a guaranteed unlearning solution, especially valuable for adhering to privacy regulations such as the GDPR and CCPA, which demand that data be forgotten upon request.

Key Features of DP2Unlearning: Privacy-Preserving: Guarantees ϵ-differential privacy for data used during training.

Efficient: Reduces computational costs significantly by avoiding retraining from scratch.

Scalable: Works well for large models and datasets, maintaining privacy without compromising performance.

Versatile: Suitable for both exact and approximate unlearning scenarios.

Experimentally Validated: DP2Unlearning outperforms existing unlearning methods in terms of model utility and forgetting effectiveness.

How DP2Unlearning Works: Stage 1: Unlearning-Ready Training

A base model is trained using ϵ-DP protection, ensuring that specific details in the training data are not memorized.

Stage 2: Pre-Unlearning Fine-Tuning

The base model is fine-tuned on unprotected data to recover its performance.

Stage 3: Unlearning Execution

The model undergoes fine-tuning on the retained data only, effectively "forgetting" the unwanted data while preserving overall performance.

Results and Evaluation: The framework has been validated through extensive experiments, demonstrating its effectiveness in achieving high model utility and strong forget quality. Compared to other unlearning techniques, DP2Unlearning provides guaranteed forgetting while maintaining or exceeding the performance of models retrained from scratch.

Future Work: Further optimizations of computational efficiency for larger models.

Investigate additional use cases for DP2Unlearning in other machine learning applications.