Entity · paper

Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning

paperactiveprovisionalsuper-tuning-from-activation-aware-pruning-to-sparse-fine-tuning-78d641a2·1 events·first seen 5h ago

Aliases: Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning

Co-occurring entities

Llama 3.2 LoRA Meta-Llama-3-8B PaFi Wanda Meta

More like this (12)

Parameter-Efficient Fine-Tuning supervised fine-tuning Layer-Adaptive Expert Pruning importance-weighted supervised fine-tuning Language Model Finetuning PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning Entropy-Aware Dense Pruning Sparse Transformer A Unifying Lens on Supervised Fine-Tuning Through Target Distribution Design OpenAI Fine-Tuning MiniMax Sparse Attention reinforcement fine-tuning

Recent events (1)

4arXiv · cs.CL·5h ago·source ↗

Super-Tuning: Pruning saliency signals repurposed for sparse parameter-efficient fine-tuning

Researchers propose Super and Supra, two sparse PEFT methods that reuse activation-weighted magnitude scores (Wanda-style) originally developed for pruning to select which parameters to update during fine-tuning. Supra combines this sparse update with LoRA under a fixed parameter budget via a budget-splitting rule. Experiments on Llama-3.2-1B and Llama-3-8B on a Math17K arithmetic task show the best Super/Supra variants outperform other tested adapter configurations. The work suggests pruning-inspired orderings are a useful, low-cost signal for identifying effective sparse fine-tuning supports.

Open Weights Progress Inference Economics Llama 3.2 Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning LoRA +4 more