technique

Language Modeling Loss

techniqueactiveprovisionallanguage-modeling-loss-b64c3b90·1 events·first seen 22d ago

Aliases: Language Modeling Loss

Co-occurring entities

knowledge distillation Weak-to-Strong Distillation LLM Pretraining

More like this (12)

Language Model Finetuning Language Model Safety Monitor unsupervised language modeling Transformer Language Models Language Models are Few-Shot Learners Arithmetic Pedagogy for Language Models Reinforcement Learning for Language Models Reasoning Language Models generative language modeling AnyLanguageModel Scaling Laws for Neural Language Models Latent Context Language Models

Recent events (1)

6arXiv · cs.LG·22d ago·source ↗

Strong Teacher Not Needed? On Distillation in LLM Pretraining

This paper challenges the conventional assumption that knowledge distillation requires a stronger teacher to produce better students. Through systematic variation of architecture sizes and training token budgets, the authors find that even small, undertrained teachers can improve larger student models when language modeling and distillation losses are properly mixed. Counterintuitively, stronger teachers can saturate or reverse distillation gains, and distillation benefits generalization more than in-domain fitting.

Training Infrastructure Frontier Model Releases knowledge distillation Language Modeling Loss Weak-to-Strong Distillation +2 more