PaFi

Super-Tuning: Pruning saliency signals repurposed for sparse parameter-efficient fine-tuning

Researchers propose Super and Supra, two sparse PEFT methods that reuse activation-weighted magnitude scores (Wanda-style) originally developed for pruning to select which parameters to update during fine-tuning. Supra combines this sparse update with LoRA under a fixed parameter budget via a budget-splitting rule. Experiments on Llama-3.2-1B and Llama-3-8B on a Math17K arithmetic task show the best Super/Supra variants outperform other tested adapter configurations. The work suggests pruning-inspired orderings are a useful, low-cost signal for identifying effective sparse fine-tuning supports.