5arXiv cs.CL (Computation and Language)·
Study disentangles task, language, and model roles in code model internal representations using circuit analysis
A new arXiv preprint investigates whether independently trained code language models converge on the same internal representations, using a 2x2 factorial design crossing Python/Rust with Qwen2.5-Coder-7B and DeepSeek-Coder-V1-6.7B. The study measures 58 Python and 57 Rust grammatical concepts via concept-circuit extraction across all four conditions. Key findings: which concepts earn dedicated circuits is consistent across models (task-determined), but where and how those circuits are organized differs substantially by model — Qwen processes concepts in late layers (~L17-19), DeepSeek in early layers (L6-7). The work reframes the 'circuit universality' question, concluding universality holds for representational content but not computational organization.
Related guides (2)
Related events (8)
Linear probes on chain-of-thought hidden states can predict reasoning non-convergence early
A new arXiv preprint studies the bimodal convergence pattern in chain-of-thought models like DeepSeek-R1-Distill-Qwen-7B, where generations either complete within a token budget (90.3% accuracy on AIME) or exhaust it without concluding (6.6% accuracy), with a 62% overall convergence rate. The authors train linear probes on hidden-state activations at early token positions (50-300) and find that layer-20 activations at token 150 achieve AUC 0.608, reliably above chance and outperforming behavioral baselines from token entropy and repetition statistics. The results suggest convergence fate is partially encoded in intermediate representations early in generation, pointing toward early-exit inference and adaptive compute allocation strategies. Statistical evidence is modest (permutation test p=0.063), limiting strong conclusions.
Evaluating Large Language Models Trained on Code
OpenAI published research on evaluating large language models trained on code, introducing the Codex model and the HumanEval benchmark for assessing code generation capabilities. The work established foundational methodology for measuring functional correctness of code produced by LLMs using a pass@k metric. This paper became a landmark reference for code-focused LLM evaluation and influenced subsequent code generation research across the field.
TypeProbe: Probing pre-trained code models for internal type representations across languages
A new arXiv preprint introduces TypeProbe, a probing study examining whether pre-trained code models internally encode formal type information in their residual streams. Using parallel Java and Python datasets, the authors find that cross-lingual type representations emerge even from untyped code, and that hidden states linearly encode result types implied by typed function application. The work is notable for targeting formal type semantics in code model interpretability, an area the authors claim has not been directly studied before.
Recursive Language Models Offer Path To Dramatically Expand Beyond the Context Window
MIT researchers Alex L. Zhang, Tim Kraska, and Omar Khattab propose Recursive Language Models (RLMs), a framework that offloads long-context processing to an external Python REPL environment, allowing models to programmatically fetch and manage text chunks via code generation. The root model spawns submodel instances to handle subtasks, aggregating their outputs recursively. Evaluated on benchmarks requiring reasoning over documents up to 11 million tokens, RLMs substantially outperform both base models and competing agentic strategies such as retrieval and summarization agents. For example, RLM-GPT-5 achieved 91.3% on BrowseComp+ versus GPT-5's inability to produce an answer, and ~50% accuracy on OOLONG-PAIRS at 1 million tokens versus near-zero for baseline approaches.
Causal circuit analysis reveals how vision-language models resolve perception-knowledge conflicts
A new arXiv preprint uses activation patching and ablation studies to identify the mechanistic basis of perception-knowledge conflict in vision-language models across three VLM families. The authors find that visual grounding is the default behavior, while knowledge-grounded responses depend on a small set of attention heads (2.5–4.8% of total) concentrated in the network's second half. Ablating these heads flips knowledge-grounded predictions to visually grounded ones in 68–96% of cases while barely affecting visually grounded predictions, revealing an asymmetric causal structure. The identified heads decompose into routing heads and writing heads, and the circuit is consistent across model families and scales.
Qwen2.5-Coder Series Open-Sourced: 32B Model Claims SOTA, Matches GPT-4o on Coding
Alibaba's Qwen team has open-sourced the Qwen2.5-Coder family of code-specialized language models, with the flagship 32B-Instruct variant claiming state-of-the-art performance among open-source code models and parity with GPT-4o on coding benchmarks. The release spans multiple model sizes, expanding on previously released smaller variants. The models are described as combining strong coding ability with general reasoning and mathematical skills.
Frontier coding agents use metaprogramming to handle esoteric programming languages
A new arXiv paper evaluates six LLM-based coding agents on four esoteric programming languages (including Brainfuck and Befunge-98), finding that the strongest agents—Claude Opus 4.6 and GPT-5.4 xhigh—often avoid writing the target language directly, instead generating it via Python metaprograms. Forbidding this strategy causes large performance drops, and text guidance alone does not transfer the capability to weaker models, though sharing Opus-derived Python helper code does sharply improve mid-tier agents. The study reveals capability stratification that mainstream benchmarks like SWE-Bench Verified compress into narrow bands, suggesting frontier agents succeed by constructing and debugging working models of unfamiliar environments rather than pattern-matching to training data.
Linear representations of grammaticality found in neural language model internals
A new arXiv preprint investigates whether grammaticality is encoded in the internal representations of neural language models, moving beyond probability-based evaluations. Using mass-mean probing, the authors find that grammatical and ungrammatical sentences are robustly separated in representational space across a wide range of pretrained models, grammatical phenomena, and languages. The encoding is not fully explained by confounds like lexical frequency or plausibility, suggesting grammaticality constitutes a coherent representational dimension. The findings contribute to debates about syntactic knowledge in LLMs and offer a probability-independent framework for evaluating grammatical competence.