technique
low-rank subspace projection
techniqueactive
low-rank-subspace-projection-d36675ca·1 events·first seen 25d agoAliases: low-rank subspace projection
Co-occurring entities
More like this (12)
Subspace Projectionlow-rank structureRank-Constrained Subspace Learning (RCSL)Recovery Subspace DimensionalityOrthogonal Residual ProjectionBoyle-Dykstra ProjectionUnstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse AutoencodersLoRA (Low-Rank Adaptation)Exact Posterior Score Estimation for Solving Linear Inverse ProblemsSparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learningrank-1 approximationintra-frame entropy-guided sparsification
Recent events (1)
Self-Policy Distillation via Capability-Selective Subspace Projection
This paper introduces Self-Policy Distillation (SPD), a self-distillation method for LLMs that requires no external signals such as correctness filters or reward models. SPD extracts a low-rank capability subspace from the model's own gradients on correctness-defining tokens, then projects KV activations into this subspace during self-generation to isolate task-relevant signal from stylistic noise. Experiments across code generation, math reasoning, and QA show up to 13% improvement over prior signal-free self-distillation methods and 15% better out-of-domain generalization.