model

Qwen2.5-1.5B-Base

modelactiveprovisionalqwen2-5-1-5b-base-aa144f71·1 events·first seen 3h ago

Aliases: Qwen2.5-1.5B-Base

Co-occurring entities

First-Token Broadcasters: Mechanistic Origins of Language Identity and Distributed Robustness in Transformers Language Identity Head Ablation Qwen2.5-7B-Instruct-1M GPT-2

More like this (12)

Qwen2.5-1.5B Qwen3.5-2B-Base Qwen2.5-0.5B Qwen3-1.7B-Base Qwen3.5-0.8B Qwen2.5-8B Qwen2.5-7B Qwen-0.5B Qwen3.5-35B-A3B-Base Qwen2.5-14B Qwen3-1.7B Qwen3-4B-Base

Recent events (1)

5arXiv · cs.CL·3h ago·source ↗

LIHA reveals first-token broadcaster heads as mechanistic source of language identity in transformers

Researchers introduce Language Identity Head Ablation (LIHA), a causal intervention that zeros individual attention heads to measure language-switching behavior across 2,700 prompt-language pairs in seven languages. Applied to GPT-2, LIHA identifies a small set of 'first-token broadcaster' heads that propagate language identity signals throughout generation, with compensatory redistribution following a hierarchical, feedforward pattern. A controlled comparison between Qwen2.5-1.5B-Base and Qwen2.5-1.5B-Instruct provides direct causal evidence that instruction tuning reorganizes language identity circuits toward early-layer localization. The findings offer mechanistic grounding for why multilingual models generate in the wrong language and why this is difficult to correct.

Evaluation and Benchmarking Alignment and RLHF First-Token Broadcasters: Mechanistic Origins of Language Identity and Distributed Robustness in Transformers Language Identity Head Ablation Qwen2.5-7B-Instruct-1M +2 more