LACUNA

LACUNA testbed introduces ground-truth parameter-level evaluation for LLM unlearning

Researchers introduce LACUNA, the first unlearning testbed with ground-truth parameter-level localization, designed to evaluate whether LLM unlearning methods truly erase knowledge from model weights or merely suppress it at the output level. The testbed injects PII of synthetic individuals into predefined parameters of 1B and 7B OLMo-based models via masked continual pretraining, enabling direct measurement of localization precision. Benchmarking current SOTA unlearning methods reveals they are highly imprecise and vulnerable to resurfacing attacks despite strong output-level performance, while successful localization enables even simple gradient-based methods to achieve robust erasure. The work addresses a critical gap in unlearning evaluation methodology relevant to privacy compliance and AI safety.