technique

FeedForward submodule

techniqueactiveprovisionalfeedforward-submodule-e0d67c5f·1 events·first seen 15d ago

Aliases: FeedForward submodule

Co-occurring entities

KV Cache SubFit Elia Cunegatti post-training compression fitted residual bypass Perplexity AI Attention submodule

More like this (12)

feed-forward transformer Feedforward Neural Networks UltraFeedback Feedback-Agent Attention submodule EmbedFilter FFR: Forward-Forward Learning for Regression Reinforcement Learning from Rich Feedback with Distributional DAgger Reinforcement Learning from Human Feedback Error Feedback (EF)Extended Kalman Filter Chain-of-Thought Fine-Tuning

Recent events (1)

6arXiv · cs.CL·15d ago·source ↗

SubFit: Submodule-Level Fitted Residual Replacement for LLM Compression

SubFit introduces a post-training LLM compression method that operates at the submodule level (Attention and FeedForward separately) rather than full layers, and selects components non-contiguously. The approach replaces removed submodules with lightweight fitted residual bypasses calibrated on small data. Evaluated across ten LLMs at sparsity levels from 12.5% to 37.5%, SubFit retains 84.6% of dense downstream accuracy at 25% sparsity versus 81.6% for the strongest baseline, while reducing perplexity degradation from 4.34x to 2.42x and delivering measurable inference speedup and KV-cache savings.

Training Infrastructure Evaluation and Benchmarking FeedForward submodule KV Cache SubFit +7 more