EmbedFilter: Using the unembedding matrix to suppress high-frequency token noise in LLM text embeddings
Researchers identify that LLM text embeddings over-express high-frequency but semantically uninformative tokens when projected onto vocabulary space, degrading embedding quality. They introduce EmbedFilter, a simple linear transformation that filters out the subspace of the unembedding matrix responsible for writing these tokens into embedding space. The method improves zero-shot performance on text embedding benchmarks across multiple LLM backbones and yields a byproduct of dimensionality reduction without quality loss. Code is publicly released.
Related guides (2)
Related events (8)
Audio-LLM-based data filtering for speech-to-speech translation via Rank-to-Distill
A new arXiv paper proposes using audio large language models to filter noisy training data for end-to-end speech-to-speech translation (S2ST). The authors introduce a two-stage Rank-to-Distill strategy: a lightweight ranker generates pseudo-labels from noisy speech pairs, which then supervise an audio-LLM to make keep/drop decisions directly from raw audio. Experiments on CVSS-C and SpeechMatrix benchmarks show up to +1.4 ASR-BLEU improvement over unfiltered baselines.
SubFit: Submodule-Level Fitted Residual Replacement for LLM Compression
SubFit introduces a post-training LLM compression method that operates at the submodule level (Attention and FeedForward separately) rather than full layers, and selects components non-contiguously. The approach replaces removed submodules with lightweight fitted residual bypasses calibrated on small data. Evaluated across ten LLMs at sparsity levels from 12.5% to 37.5%, SubFit retains 84.6% of dense downstream accuracy at 25% sparsity versus 81.6% for the strongest baseline, while reducing perplexity degradation from 4.34x to 2.42x and delivering measurable inference speedup and KV-cache savings.
Make LLM Fine-tuning 2x faster with Unsloth and 🤗 TRL
Hugging Face published a blog post detailing an integration between Unsloth and TRL (Transformer Reinforcement Learning) library that claims to achieve 2x faster LLM fine-tuning. The post covers how Unsloth optimizes training kernels to reduce memory usage and increase throughput. This is relevant to practitioners looking to reduce compute costs and time for fine-tuning large language models.
Hyperfitting Explained: Terminal Geometric Expansion in Final Transformer Layers Drives Diversity Gains
This paper investigates the 'hyperfitting' phenomenon—where fine-tuning LLMs to near-zero loss on small datasets improves open-ended generation and reduces repetition—and demonstrates it is mechanistically distinct from temperature scaling. Entropy-matched control experiments falsify both the temperature-equivalence and static vocabulary reweighting hypotheses, instead localizing the effect to a 'Terminal Expansion' in the final transformer block where feature-space dimensionality expands by ~80.8 dimensions, enabling promotion of deep-tail tokens via context-dependent rank reordering. The authors introduce Late-Stage LoRA, a targeted fine-tuning strategy updating only the final 5 layers, achieving robust generation with minimal parameter updates.
Text Analytics Evaluation Framework: Benchmarking LLMs on Social Media NLP Tasks
Researchers introduce a 470-question evaluation framework to assess LLM performance on aggregated social media text, applied to Twitter datasets across sentiment analysis, hate speech detection, and emotion recognition. Results show performance degrades substantially as input scale exceeds 500 instances, particularly for open-weights models on numerical tasks. Multi-label and target-dependent scenarios also show notable performance drops, and task complexity progressively erodes accuracy from basic semantic identification to comparison and counting operations. The findings point to architectural bottlenecks in current LLMs for rigorous quantitative analysis over large text collections.
ATWU: Token-level importance learning improves LLM unlearning via retain-conflict criterion
This paper introduces Alternating Token-Weighted Unlearning (ATWU), a framework that learns which tokens in a forget sample are most relevant to unlearning by characterizing their conflict with the retain objective. Rather than relying on auxiliary models or heuristics, ATWU jointly learns token forget-specificity and model parameters using a lightweight linear scorer over hidden states. Evaluated on TOFU and RWKU benchmarks, ATWU achieves state-of-the-art forget-retain trade-offs and produces token-level scores that align with ground-truth forget-specific spans.
MLSkip: Data skipping for ML filter predicates using Parquet metadata and neural network verification
MLSkip introduces data skipping techniques for ML-based filter predicates in databases, a problem not addressed by traditional min-max pruning methods. The approach leverages Parquet's existing min-max metadata combined with neural network verification techniques to prune non-qualifying row groups. On TPC-H and TPC-DS benchmarks with ReLU architectures, the method achieves 27.4% average pruning effectiveness for low-selectivity filters, improving to 38.31% with a proposed 2D convex hull metadata structure, yielding a 1.07× end-to-end speedup in DuckDB over PyTorch.
Word Coverage Score (WCS): Measuring Lexical Suppression from LLM Sampling Filters
This paper introduces the Word Coverage Score (WCS), a metric that quantifies how much contextually appropriate low-frequency vocabulary is pruned away by standard sampling strategies (Top-p, Top-k, Min-p) in LLMs. The authors audit open-weight models against human-authored corpora to measure the 'lexical survival rate' of high-information words under typical decoding defaults. Their findings provide quantitative evidence that industry-standard sampling parameters act as unintended censorship mechanisms, suppressing linguistic diversity even when rare words exist within the model's probability distribution. The WCS is proposed as a diagnostic tool for tuning the coherence–lexical-richness trade-off.