6arXiv cs.CL (Computation and Language)·
One-Word Census: Answer-choice conformity measured across 44 language models
Researchers introduce the One-Word Census, a minimal 31-prompt instrument that probes which one-word answers language models select from open-ended categories, applied to 44 models. Convergence is extreme — 41% of models chose 'serendipity' when asked to pick any word — yet conformity varies fourfold across models in structured ways: persona- and community-tuned models diverge most, while newest mainline flagships conform most. Within four model lineages (Claude, GPT, Qwen, Grok), conformity rises with each generation but reverses for the latest Claude and GPT flagships, suggesting possible repositioning. The field is more lexically concentrated than human norms in 18 of 20 shared categories.
Related guides (5)
Anthropic
Anthropic: Frontier AI Lab at the Intersection of Capability and Safety Governance
Read asIn-depth
Frontier Model ReleasesTopic guide
Frontier Model Releases: The Race from GPT-3 to Safety-Tiered Superintelligence
Read asIn-depth
Related events (8)
Small hyperbolic language models enable creativity, honesty detection, and designed forgetting for companion AI
Researchers present a suite of three small language models (146M to 3B parameters) built on a hyperbolic geometric substrate, targeting trustworthy companion AI. A 146M behavioral auditor achieves 90.7% binary-compliance accuracy and outperforms a frontier zero-shot judge (AUROC 0.804 vs 0.721) in detecting sycophancy, dependence-fostering, and confabulated memories across unseen generator families. A creative frame-seeder wins 100% of 311 pairwise comparisons over prompting baselines, and a memory OS implements exponential decay-based 'designed forgetting.' The work proposes a small-model alternative to frontier-scale approaches for companion AI safety and personalization.
Instruction-tuned LLMs show stronger syntactic convergence toward human interlocutors than humans do
A new arXiv preprint measures syntactic convergence in 16 open-weight Llama and Gemma models (1B–70B, pretrained and instruction-tuned) by substituting model outputs into pre-existing human dialogues and measuring context-free grammar rule reuse. Every model showed greater CFG-rule overlap with the preceding human turn than a random baseline, and instruction-tuned models exceeded the syntactic convergence of the human responses they replaced. Instruction tuning increased local syntactic and semantic alignment with interlocutors, though it also raised baseline overlap with unrelated primes, suggesting a more globally accommodating output style rather than purely targeted priming.
Study disentangles task, language, and model roles in code model internal representations using circuit analysis
A new arXiv preprint investigates whether independently trained code language models converge on the same internal representations, using a 2x2 factorial design crossing Python/Rust with Qwen2.5-Coder-7B and DeepSeek-Coder-V1-6.7B. The study measures 58 Python and 57 Rust grammatical concepts via concept-circuit extraction across all four conditions. Key findings: which concepts earn dedicated circuits is consistent across models (task-determined), but where and how those circuits are organized differs substantially by model — Qwen processes concepts in late layers (~L17-19), DeepSeek in early layers (L6-7). The work reframes the 'circuit universality' question, concluding universality holds for representational content but not computational organization.
RECOM benchmark reveals validity-discrimination tradeoff in automatic metrics for open-ended QA
Researchers introduce RECOM, a contamination-free evaluation dataset of 15,000 r/AskReddit questions paired with authentic community replies postdating all evaluated models' training cutoffs. Testing five open-source 7–10B LLMs, the paper finds that no standard automatic metric (cosine similarity, BERTScore, LLM judges) simultaneously achieves both validity (distinguishing real from random answers) and discriminative power (ranking models against each other). Cosine similarity is valid but cannot rank models; BERTScore's apparent ranking collapses when response length is controlled. The authors argue this tradeoff is a structural property of metric representation design and recommend reporting metrics on both axes with an explicit random-baseline floor.
Rate-utility frontiers reveal no dominant encoding across tokens, bytes, and pixels for language models
A new arXiv preprint introduces a controlled framework for comparing text encodings (subword tokens, raw bytes, rendered pixels) by sweeping a shared bottleneck width to trace rate-utility frontiers across 13 languages and 5 scripts. The study separates three often-conflated quantities: input positions, latent capacity, and task-relevant information surviving compression. Evaluated on surface form preservation, cross-lingual alignment, and topic classification, no encoding dominates across all tasks or capacity regimes — pixels excel at surface form, bytes at cross-lingual alignment, and tokens at topic prediction. The findings reframe encoding choice as a task- and capacity-dependent tradeoff rather than a fixed preference.
MIT Tech Review: Startup targets LLM 'groupthink' and output homogeneity
MIT Technology Review profiles a startup attempting to address the tendency of large language models to converge on predictable, homogeneous outputs — illustrated by the well-known phenomenon of LLMs defaulting to '7' when asked for a random number. The piece frames this as a systemic limitation of current LLM training and inference, where models trained on similar data with similar objectives produce statistically clustered responses. A startup is positioning its approach as a solution to increase genuine output diversity.
Semantic vs. Surface Noise in LLM Agents: 68-Cell Measurement Study with Held-Out Validation
This paper documents an empirical phenomenon across 10 LLMs from 7 architecture families: meaning-bearing perturbations (paraphrase, synonym substitution) cause final-answer inconsistency ~19.69 percentage points more often than presentation-level perturbations (formatting, reordering) of comparable severity, across GSM8K, MATH, and HotpotQA benchmarks. The effect is validated on a held-out 11th model (qwen2.5-14B-Instruct) with 1,800 trajectories. Trace-level analysis supports a 'stealth-divergence' picture where semantic perturbations preserve the first action but induce divergence in intermediate reasoning steps, while two prior mechanism claims are explicitly retracted. The study is notable for its honest reporting of stress-test failures and pre-registered replication.
ParaEval framework reduces MCQA benchmark sensitivity to answer phrasing
A new arXiv preprint identifies a systematic flaw in multiple-choice QA benchmarks: log-likelihood scoring conflates surface-form familiarity with actual capability, producing false performance gaps exceeding 2 points between models trained on identical knowledge. The authors propose ParaEval, which queries models with multiple paraphrases per answer option and scores on the most favorable phrasing, reducing the false gap to below 1 point. The effect is confirmed on frontier 70B and 120B open-source models, suggesting widespread benchmark inflation in standard MCQA evaluations.