Entity · technique

Adaptive Depth Sparse Framework

techniqueactiveprovisionaladaptive-depth-sparse-framework-1a417faa·1 events·first seen 3d ago

Aliases: Adaptive Depth Sparse Framework

Co-occurring entities

More like this (12)

Graph Sparse Sampling DeepSeek Sparse Attention MiniMax Sparse Attention Block Sparse Attention Advanced AI Scaling Framework Sparse Autoencoder AdaJEPA: An Adaptive Latent World Model Cross-Layer Sparse Attention DeepScaleR ProbSparse Attention block sparse matrices sparse frame sampling

Recent events (1)

5arXiv · cs.CL·3d ago·source ↗

AdaDSF: Adaptive Depth Sparse Framework for inference-efficient LLMs without retraining

Researchers introduce AdaDSF, a method that converts pre-trained LLMs into depth-sparse models without full retraining by using cosine similarity between layer inputs and outputs to identify redundant computation. A lightweight router selects informative tokens per layer, and a feature-preserving alignment objective maintains output quality. Evaluated on GPT-NeoX and Qwen2.5, AdaDSF reduces inference FLOPs while outperforming baselines including MoD, D-LLM, and DLO under comparable sparsity budgets.

Inference Economics Qwen2.5 GPT-NeoX Adaptive Depth Sparse Framework +1 more