technique

temperature scaling

techniqueactivetemperature-scaling-9f6e8216·1 events·first seen 25d ago

Aliases: temperature scaling

Co-occurring entities

Terminal Expansion large language models LoRA Hyperfitting Late-Stage LoRA

More like this (12)

power-law scaling test-time compute scaling Responsible Scaling Policy Shannon Scaling Law on-device MoE scaling law inference-time compute scaling gradient compression warmdown learning-rate schedule AI scaling laws SDE (Stochastic Differential Equation LR scaling)Anthropic Responsible Scaling Policy Conditional Scale Entropy

Recent events (1)

6arXiv · cs.CL·25d ago·source ↗

Hyperfitting Explained: Terminal Geometric Expansion in Final Transformer Layers Drives Diversity Gains

This paper investigates the 'hyperfitting' phenomenon—where fine-tuning LLMs to near-zero loss on small datasets improves open-ended generation and reduces repetition—and demonstrates it is mechanistically distinct from temperature scaling. Entropy-matched control experiments falsify both the temperature-equivalence and static vocabulary reweighting hypotheses, instead localizing the effect to a 'Terminal Expansion' in the final transformer block where feature-space dimensionality expands by ~80.8 dimensions, enabling promotion of deep-tail tokens via context-dependent rank reordering. The authors introduce Late-Stage LoRA, a targeted fine-tuning strategy updating only the final 5 layers, achieving robust generation with minimal parameter updates.

Inference Economics Alignment and RLHF Terminal Expansion large language models temperature scaling +3 more