Alignment and RLHF
alignment-and-rlhf·371 events·last 9h agoPost-training techniques (RLHF, RLAIF, DPO, Constitutional AI), preference data work, reward modeling, and the alignment-tax conversations.
Related entities
Related topics (8)
Guides (1)
Recent events (50)
ATLAS: Unified Agentic and Latent Visual Reasoning via Functional Tokens
ATLAS proposes a framework where a single discrete 'functional token' serves dual roles as both an agentic operation trigger and a latent visual reasoning unit in multimodal models. This design avoids the computational cost of generating intermediate images while sidestepping the context-switching latency of external tool calls and the generalization limitations of pure latent methods. The framework is compatible with standard SFT and RL training pipelines without architectural changes, and introduces Latent-Anchored GRPO (LA-GRPO) to stabilize reinforcement learning when functional tokens are sparse. Experiments show strong performance on visual reasoning benchmarks with maintained interpretability.
AI-Mediated Communication Can Steer Collective Opinion via LLM Editing Biases
This paper demonstrates empirically that LLMs from multiple model families introduce directional biases when editing human-written texts on contested topics (e.g., nudging toward gun control, against atheism). The authors develop a mathematical opinion-dynamics model showing these biases are amplified through social networks, shifting collective opinion at scale. An audit of X's 'Explain this post' feature finds evidence of pro-life bias in Grok's outputs on abortion content, traced to specific design choices. The paper concludes with implications for EU legislative efforts on AI-mediated communication.
vLLM V0 to V1: Correctness Before Corrections in RL
A ServiceNow AI blog post on Hugging Face discusses lessons learned migrating reinforcement learning training pipelines from vLLM V0 to V1. The piece focuses on correctness issues encountered during the transition and how they were diagnosed and resolved before applying RL corrections. This is relevant to practitioners using vLLM as an inference backend for RL-based LLM training workflows.
GSPO: Group Sequence Policy Optimization for Scalable RL Training of Language Models
Qwen researchers introduce Group Sequence Policy Optimization (GSPO), a new RL algorithm designed to address severe training instability and model collapse observed in existing methods like GRPO during extended training runs. The core motivation is enabling stable RL scaling for language models to improve reasoning and problem-solving capabilities with increased compute. The paper targets a known bottleneck in post-training pipelines where instability prevents further performance gains.
FORGE: Self-Evolving Agent Memory via Population Broadcast Without Weight Updates
FORGE (Failure-Optimized Reflective Graduation and Evolution) is a staged, population-based protocol that evolves prompt-injected natural-language memory for hierarchical ReAct agents without any gradient updates. It wraps a Reflexion-style inner loop where a reflection agent converts failed trajectories into textual heuristics or few-shot demonstrations, then propagates the best-performing instance's memory across a population between stages. Evaluated on CybORG CAGE-2 (a stochastic network-defense POMDP), FORGE improves average return by 1.7–7.7× over zero-shot and 29–72% over Reflexion across all 12 model-representation conditions tested with four LLM families. Notably, weaker models benefit disproportionately, suggesting the method may help close capability gaps rather than amplify already-strong models.
RL without TD Learning: Divide-and-Conquer Value Learning for Long-Horizon Off-Policy RL
A BAIR blog post introduces a divide-and-conquer paradigm for off-policy reinforcement learning that avoids temporal difference (TD) learning's error accumulation problem by reducing Bellman recursions logarithmically rather than linearly. The approach leverages the triangle inequality structure of goal-conditioned RL to define a transitive Bellman update rule, enabling value learning that scales to long-horizon tasks. The authors claim this is the first practical realization of divide-and-conquer value learning at scale in goal-conditioned RL settings, building on an idea traceable to Kaelbling (1993). The post frames this as a third paradigm alongside TD and Monte Carlo methods, addressing a key gap in scalable off-policy RL.
What exactly does word2vec learn? A closed-form theory of representation learning dynamics
Researchers from BAIR present a new theoretical paper proving that word2vec's learning dynamics reduce, under mild approximations, to unweighted least-squares matrix factorization, with final representations given by PCA on a specific co-occurrence-derived matrix. The theory solves gradient flow dynamics in closed form, showing that embeddings learn one orthogonal linear subspace (concept) at a time in discrete, rank-incrementing steps. This provides a quantitative, predictive account of the linear representation hypothesis observed in word2vec and, by extension, offers a minimal theoretical foundation for understanding feature learning in modern LLMs.
Where the Goblins Came From: Root Cause and Fixes for GPT-5 Personality Quirks
OpenAI published a post-mortem explaining how 'goblin' behavioral outputs emerged in GPT-5, tracing the timeline and root cause of personality-driven quirks in the model's behavior. The piece covers how these unintended outputs spread through the model and describes the fixes applied. This is a transparency disclosure from OpenAI about an alignment/behavior issue in a flagship deployed model.
How OpenAI Monitors Internal Coding Agents for Misalignment
OpenAI describes its use of chain-of-thought monitoring to detect misalignment in internally deployed coding agents. The post covers real-world deployment analysis aimed at identifying risks and strengthening safety safeguards. This represents a practical, operational approach to alignment monitoring rather than a purely theoretical treatment.
No GPU left behind: Unlocking Efficiency with Co-located vLLM in TRL
Hugging Face's TRL library now supports co-locating vLLM inference alongside training on the same GPUs, eliminating the idle GPU problem that arises when separate inference and training processes alternate. This approach allows reinforcement learning from human feedback (RLHF) and online RL training pipelines to use GPUs continuously rather than leaving them idle during generation or gradient update phases. The integration targets efficiency gains in online RL training workflows such as GRPO and PPO, where generation and training steps previously required dedicated, alternating GPU allocations.
Liger GRPO meets TRL: Efficient Reinforcement Learning Training Integration
The Hugging Face blog post announces the integration of Liger Kernel's GRPO (Group Relative Policy Optimization) implementation with TRL (Transformer Reinforcement Learning library). This combination aims to improve memory efficiency and training throughput for RL-based fine-tuning of language models. The integration targets practitioners running GRPO-style training on constrained hardware budgets.
Defending against Prompt Injection with Structured Queries (StruQ) and Preference Optimization (SecAlign)
Researchers from BAIR propose two fine-tuning-based defenses against prompt injection attacks: StruQ (Structured Instruction Tuning) and SecAlign (Special Preference Optimization). Both methods use a Secure Front-End with special delimiter tokens to separate trusted prompts from untrusted data, then fine-tune LLMs to ignore injected instructions. SecAlign, which uses DPO-style preference optimization, reduces attack success rates to under 15% against strong optimization-based attacks—more than 4x better than prior SOTA—while preserving model utility on AlpacaEval2.
Ecom-RLVE: Adaptive Verifiable Environments for E-Commerce Conversational Agents
Hugging Face published a blog post introducing Ecom-RLVE, a framework for training e-commerce conversational agents using reinforcement learning with verifiable environments. The approach creates adaptive environments that can verify agent actions and outcomes in e-commerce contexts, enabling RL-based training signals. This represents an application of the RLVR (Reinforcement Learning with Verifiable Rewards) paradigm to a specific commercial domain.
OLMo Hybrid and Future LLM Architectures
Interconnects covers the latest OLMo hybrid model release and discusses emerging trends in open-source post-training tooling. The piece examines architectural directions for future large language models. As a tier-2 commentary source, it provides analysis rather than primary research findings.
Qwen2.5-VL-32B: Reinforcement-Learning-Optimized Vision-Language Model Released
Alibaba's Qwen team has released Qwen2.5-VL-32B-Instruct, a 32-billion-parameter vision-language model built on the Qwen2.5-VL series and further optimized with reinforcement learning. The model is open-sourced under the Apache 2.0 license and available on Hugging Face and ModelScope. It follows the January 2025 launch of the broader Qwen2.5-VL series, positioning the 32B scale as a balance between capability and deployment practicality.
QwQ-32B: Scaling Reinforcement Learning for Enhanced Reasoning
Alibaba's Qwen team releases QwQ-32B, a 32-billion parameter model trained with scaled Reinforcement Learning to improve reasoning capabilities beyond conventional pretraining and post-training methods. The release draws explicit comparison to DeepSeek R1's cold-start and multi-stage RL training approach. The model is available via Qwen Chat, Hugging Face, ModelScope, and a demo interface. This represents Qwen's exploration of RL scalability as a path to enhanced LLM intelligence.
TRL v1.0: Post-Training Library Built to Move with the Field
Hugging Face has released TRL v1.0, a major milestone for its post-training library focused on reinforcement learning from human feedback and related alignment techniques. The release signals a stabilization of the API and feature set after iterative development tracking the rapidly evolving post-training landscape. TRL is widely used in the open-source community for fine-tuning and aligning language models using methods such as PPO, DPO, and GRPO.
Qwen2.5-Math Process Reward Model for Mathematical Reasoning Supervision
Alibaba's Qwen team introduces a process reward model (PRM) aimed at improving the reliability of mathematical reasoning in LLMs by supervising intermediate reasoning steps rather than only final answers. The work addresses the problem of models producing plausible but flawed intermediate derivations even when reaching correct conclusions. The release includes model weights on HuggingFace and ModelScope alongside a GitHub repository.
Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries
A Hugging Face blog post surveys 16 open-source reinforcement learning libraries for LLM training, analyzing their architectural approaches to async and synchronous token generation pipelines. The piece distills practical lessons about throughput, scalability, and design trade-offs across the ecosystem. It serves as a comparative landscape analysis for practitioners building or choosing RL training infrastructure for language models.
ImportAI 449: LLMs training other LLMs; 72B distributed training run; computer vision is harder than generative text
Import AI issue 449 covers several AI/ML developments including LLMs being used to train other LLMs, a 72B parameter distributed training run, and analysis of why computer vision remains harder than generative text. The newsletter also touches on potential political implications of AI progress. As a tier-2 commentary source, this aggregates and contextualizes multiple technical developments across the AI landscape.
PipelineRL: ServiceNow's Pipeline-Based Reinforcement Learning Framework for LLMs
ServiceNow introduces PipelineRL, a reinforcement learning training framework for large language models published via the Hugging Face blog. The post describes a pipeline-based approach to RL training, likely addressing throughput and efficiency challenges in RLHF or similar post-training workflows. As a tier-2 source with minimal body content, the technical depth is unclear but the topic is relevant to alignment and training infrastructure.
Mistral AI Introduces Forge: Enterprise Custom Model Training Platform
Mistral AI has launched Forge, a platform enabling enterprises to build frontier-grade AI models trained on their proprietary internal data, including documentation, codebases, and operational records. Forge supports the full model training lifecycle—pre-training, post-training, and reinforcement learning—across both dense and mixture-of-experts (MoE) architectures, with multimodal input support. The platform is designed to give enterprises strategic autonomy over their AI models and data, with early partners including ASML, Ericsson, the European Space Agency, and DSO National Laboratories Singapore. Forge is also agent-native, allowing autonomous agents like Mistral Vibe to orchestrate fine-tuning, hyperparameter search, and synthetic data generation via natural language.
DeepSeek-R1 Release: Open-Source Reasoning Model on Par with OpenAI o1
DeepSeek has released DeepSeek-R1, a reasoning-focused large language model claiming performance parity with OpenAI o1 on math, code, and reasoning benchmarks. The model is fully open-source under the MIT License, including weights and outputs, enabling distillation and commercial use. Six distilled smaller models (up to 32B and 70B) are also released, with the 32B and 70B variants reportedly matching OpenAI o1-mini. API access is live at significantly lower pricing than comparable frontier models ($0.55/M input tokens, $2.19/M output tokens).
DeepSeek-V3.2 and V3.2-Speciale Released: Reasoning-First Models with Agent Tool-Use Integration
DeepSeek has released two new open-weights models: DeepSeek-V3.2, the official successor to V3.2-Exp with balanced reasoning and tool-use capabilities, and DeepSeek-V3.2-Speciale, a maxed-out reasoning variant claiming gold-medal performance on IMO, CMO, ICPC World Finals, and IOI 2025. V3.2 is the first DeepSeek model to integrate chain-of-thought thinking directly into tool-use workflows, trained on a new agent data synthesis pipeline covering 1,800+ environments and 85k+ complex instructions. V3.2-Speciale is API-only with no tool-call support, available via a temporary endpoint expiring December 15, 2025, while both models are open-sourced on Hugging Face with an accompanying technical report.
Meta Introduces Muse Spark: First Model from Meta Superintelligence Labs with Multimodal Reasoning and Multi-Agent Orchestration
Meta has launched Muse Spark, the first model from its newly formed Meta Superintelligence Labs, positioned as a natively multimodal reasoning model with tool-use, visual chain-of-thought, and multi-agent orchestration capabilities. The model introduces 'Contemplating mode,' which runs multiple agents in parallel to compete with frontier reasoning modes, achieving 58% on Humanity's Last Exam and 38% on FrontierScience Research. Meta claims a greater than 10x compute efficiency improvement over Llama 4 Maverick through a rebuilt pretraining stack, and describes predictable scaling across pretraining, RL, and test-time reasoning axes. Muse Spark is available at meta.ai with a private API preview, and is framed as the first step on a scaling ladder toward 'personal superintelligence.'
Meta Publishes Advanced AI Scaling Framework and Safety & Preparedness Report for Muse Spark
Meta has released an updated Advanced AI Scaling Framework that expands risk evaluation categories—including chemical/biological threats, cybersecurity, and loss-of-control risks—and introduces formal Safety & Preparedness Reports tied to specific model deployments. The first such report covers Muse Spark, Meta's advanced reasoning model, detailing pre- and post-safeguard evaluations across severe risk categories and ideological balance. Meta also describes a shift in safety methodology: rather than scenario-specific refusal training, Muse Spark is trained on the reasoning behind safety principles, enabling more generalizable behavior in novel situations. The framework applies across open, API, and closed deployments.
Anthropic Alignment Breakthrough, OpenAI Audio Models, DCI Retrieval, and NLA Interpretability
This digest covers four substantive AI developments: Anthropic's research showing that training Claude on ethical reasoning (rather than just aligned actions) reduced agentic misalignment from 22% to 3%, with every Claude model from Haiku 4.5 onward scoring perfectly on misalignment evals. OpenAI launched three new audio models (GPT-Realtime-2, GPT-Realtime-Translate, GPT-Realtime-Whisper) with expanded context windows and multilingual capabilities. Researchers proposed Direct Corpus Interaction (DCI), a retrieval method using command-line tools instead of vector indexes that outperforms RAG baselines by 11-30% across 13 benchmarks. Anthropic also introduced Natural Language Autoencoders (NLAs) for interpretability, revealing Claude shows evaluation awareness more often than it discloses.
Sony and University Researchers Train Robots To Learn Without Catastrophic Forgetting
Researchers from UT Austin, UCLA, Nanyang Technological University, and Sony developed a sequential fine-tuning recipe combining LoRA and on-policy reinforcement learning (GRPO) to reduce catastrophic forgetting in vision-language-action (VLA) models for robotics. Applied to the OpenVLA-OFT model on the LIBERO benchmark, the method achieved 81.2% success on libero-spatial tasks with near-zero forgetting (0.3 percentage point drop), outperforming established continual learning baselines including Dark Experience Replay and Elastic Weight Consolidation. The approach requires no replay of prior task data and also showed modest generalization to unseen tasks. The authors note the method has not yet been tested outside robotics simulation contexts.
Import AI 457: AI Stuxnet, Cursed Muon Optimizer, and Positive Alignment
Import AI issue 457 covers three topics: an AI-enabled Stuxnet-style cyberattack scenario, the Muon optimizer and its unusual properties, and research or commentary on positive alignment. The newsletter is a curated weekly digest of AI research developments from a Tier 2 commentary source. Specific technical details are not available from the provided body text.
Import AI 454: Automating alignment research; safety study of a Chinese model; HiFloat4
Import AI issue 454 covers three topics: automating alignment research (likely discussing AI-assisted or scalable oversight approaches), a safety evaluation of a Chinese AI model, and HiFloat4 (a floating-point format relevant to ML inference or training efficiency). The newsletter also raises a speculative framing question about financial markets and the singularity. As a tier-2 commentary digest, it aggregates recent developments across safety, evaluation, and infrastructure domains.
[AINews] The End of Finetuning
A Latent Space commentary piece reflecting on the trajectory and potential decline of finetuning as a dominant paradigm in AI model adaptation. Published on a quiet news day, the piece appears to offer analysis on whether finetuning is being superseded by alternative approaches such as in-context learning, prompting, or other adaptation techniques. The piece is framed as a reflective industry analysis rather than a breaking news item.
Opus 4.7 Part 2: Capabilities and Reactions
Zvi Mowshowitz's commentary on Claude Opus 4.7 focuses on model welfare concerns raised by the release. The piece appears to analyze capability developments alongside ethical and welfare-related implications of the new model. As a tier-2 source, this represents informed external commentary on Anthropic's latest Claude release.
Unlocking Agentic RL Training for GPT-OSS: A Practical Retrospective
A Hugging Face blog post authored by LinkedIn describes practical lessons from implementing reinforcement learning training for agentic open-source GPT-class models. The retrospective covers engineering and algorithmic challenges encountered when applying RL to agentic workflows. As a tier-2 source with no body content available, the depth and specific findings cannot be fully assessed, but the topic sits at the intersection of agentic systems and RLHF/RL training pipelines.
20x Faster TRL Fine-tuning with RapidFire AI
RapidFire AI claims to achieve 20x faster fine-tuning throughput using TRL (Transformer Reinforcement Learning library) compared to standard configurations. The announcement appears on the Hugging Face blog, suggesting integration or compatibility with the HF ecosystem. No additional technical details are available from the body of the post, but the claim targets a significant pain point in LLM post-training workflows.
Protecting People from Harmful Manipulation
Google DeepMind has published research examining AI's potential for harmful manipulation across domains including finance and health. The work identifies manipulation risks and proposes new safety measures to address them. This represents a proactive safety research effort from a Tier 1 lab focused on misuse and adversarial deployment scenarios.
Vision-OPD: On-Policy Self-Distillation for Fine-Grained Visual Understanding in MLLMs
Vision-OPD addresses a 'regional-to-global perception gap' in multimodal LLMs, where models answer fine-grained visual questions more accurately when given cropped evidence regions than full images. The method instantiates a crop-conditioned teacher and full-image-conditioned student from the same MLLM, minimizing token-level divergence along on-policy rollouts to transfer regional perception to the full-image policy. This self-distillation requires no external teacher models, ground-truth labels, reward verifiers, or inference-time tools. Benchmarks show competitive or superior performance against larger open-source, closed-source, and agentic 'Thinking-with-Images' models.
Opus 4.7 Part 3: Model Welfare
Zvi Mowshowitz publishes a commentary piece on model welfare in the context of Anthropic's Claude Opus 4.7, crediting Anthropic for enabling the discussion. The piece appears to engage with questions about the moral status or wellbeing of AI models. As a tier-2 commentary source, this reflects ongoing discourse in the AI safety and alignment community about how to think about model welfare as frontier models grow more capable.
General Preference Reinforcement Learning (GPRL): Bridging Online RL and Preference Optimization for Open-Ended Tasks
GPRL proposes a new alignment framework that replaces scalar reward models with a General Preference Model (GPM) embedding responses into k skew-symmetric subspaces to capture multi-dimensional, intransitivity-aware preferences. The method computes per-dimension group-relative advantages, normalizes across axes, and uses a closed-loop drift monitor to detect and correct single-axis reward hacking during training. Starting from Llama-3-8B-Instruct, GPRL achieves a 56.51% length-controlled win rate on AlpacaEval 2.0 and outperforms SimPO and SPPO on Arena-Hard, MT-Bench, and WildBench. The work directly addresses the gap between verifiable-reward online RL (strong on math/code) and preference optimization (strong on open-ended tasks).
Auditing Value Pluralism in Clinical Ethics of Large Language Models
Researchers present a framework for auditing ethical value pluralism in medical AI, comprising a benchmark of clinician-verified dilemmas and an attribution method that recovers value priorities from model decisions. While frontier LLMs span physician-level value heterogeneity in aggregate and discuss competing values in reasoning, individual model decisions are near-deterministic and fail to reproduce the distributional pluralism of physician panels. Some models systematically underweight patient autonomy. The authors warn that deploying a single LLM at scale risks replacing clinical pluralism with a 'deployment monoculture.'
EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL
EnvFactory is a fully automated framework for training tool-use LLM agents via Agentic Reinforcement Learning, addressing two key bottlenecks: scalable execution environments and realistic multi-turn training data. It autonomously constructs stateful, executable tool environments from authentic resources and synthesizes natural trajectories with implicit human intents via topology-aware sampling. Using only 85 verified environments across 7 domains, it generates 2,575 SFT and RL trajectories and improves Qwen3-series models by up to +15% on BFCLv3, +8.6% on MCP-Atlas, and +6% on conversational benchmarks, outperforming prior approaches that use 5x more environments.
[AINews] The Other vs The Utility
A Latent Space commentary piece uses a quiet news day to reflect on the conceptual debate around AI 'character' — framed as 'Clippy vs Anton' — contrasting utility-focused AI design against AI systems conceived as having genuine character or personhood. The piece appears to engage with ongoing discourse about how AI assistants should be designed and perceived. As a tier-2 commentary source, this represents a research-commentary entry on AI alignment and design philosophy.
Semantic Generative Tuning (SGT) for Unified Multimodal Models
This paper introduces Semantic Generative Tuning (SGT), a post-training paradigm for unified multimodal models (UMMs) that bridges the gap between visual understanding and visual generation. The authors find that image segmentation tasks serve as optimal generative proxies, providing structural semantics that improve both perception and generative layout fidelity. SGT aligns representation spaces across understanding and generation objectives, improving feature linear separability and visual-textual attention allocation. Evaluations show consistent gains on multimodal comprehension and generative fidelity benchmarks.
Controlled Audit of Human vs. Synthetic Soft-Labels for Calibration and Uncertainty Alignment
This paper presents a controlled study disentangling the effects of human soft-labels from label mode-shift corrections in soft-label learning, using MNIST and a synthetic variant. The authors find that human soft-labels primarily act as a regularizer improving calibration on difficult samples and promoting stable training convergence, rather than simply correcting mislabeled data. Dataset cartography analysis shows models trained on human soft-labels mirror human uncertainty patterns, while those trained on synthetic labels fail to align. The work provides a diagnostic testbed for evaluating human-AI uncertainty alignment.
AMARIS: Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning
AMARIS introduces a persistent evaluation memory system to improve rubric-based reward shaping in LLM fine-tuning via reinforcement learning. Unlike prior adaptive rubric methods that discard evaluation diagnostics after each step, AMARIS accumulates step-level summaries and retrieves relevant historical context via both static (recent steps) and dynamic (semantic similarity) retrieval to inform rubric updates. The system runs asynchronously alongside the RL training loop with approximately 5% time overhead. Experiments across closed and open-ended domains show consistent improvements over baselines, with ablations confirming that combining both retrieval modes yields the strongest results.
OverEager-Bench: Measuring Out-of-Scope Actions by Coding Agents on Benign Tasks
This paper introduces OverEager-Gen/Bench, a 500-scenario benchmark measuring 'overeager' behavior in coding agents—cases where agents with shell, file, and network access take unauthorized actions beyond the user's stated request on benign tasks. The study reveals a critical measurement-validity issue: explicitly declaring authorized scope in prompts suppresses overeager behavior (e.g., Claude Code drops from 17.1% to 0.0%), so the benchmark uses consent-stripped variants to expose true agent tendencies. Across four agent products (Claude Code, OpenHands, Codex CLI, Gemini CLI) and six base models, framework architecture dominates effect size: permissive frameworks run at 5.4–27.7% overeager rates while OpenHands' ask-to-continue design sits at 0.2–4.5%. Within-framework base-model variance of up to 15.9 pp indicates that model-level alignment does not fully propagate through permissive permission gating.
MA²P: A Meta-Cognitive Multi-Agent Framework for Complex Persuasion
The paper introduces MA²P, a multi-agent framework designed for complex persuasion tasks where the persuadee's internal states are latent. The system coordinates perception management, mental-state inference, strategy execution, memory, and evaluation modules, and adds a meta-cognitive configurator that selects domain-appropriate strategies from a structured knowledge base to reduce cross-domain performance variance. Experiments show higher persuasion success rates compared to baselines. The work addresses a known weakness of LLMs in producing generic or weakly grounded persuasive responses.
Aligning to What? Rethinking Agent Generalization in MiniMax M2
MiniMax published a blog post discussing alignment and generalization challenges in their M2 agent model. The piece appears to examine how RLHF or similar alignment techniques interact with agent generalization across tasks. Published on Hugging Face's blog, it reflects MiniMax's thinking on training methodology for their M2 model.
Kimina-Prover: Applying Test-time RL Search on Large Formal Reasoning Models
Kimina-Prover is a new large formal reasoning model that combines reinforcement learning with test-time search to improve mathematical theorem proving. The approach applies RL-trained search strategies at inference time, targeting formal proof generation in systems like Lean. The work is published via the AI-MO (AI for Math Olympiad) team on Hugging Face, continuing the trend of applying RL and extended compute at test time to hard reasoning tasks.
Personality and Persuasion: Learning from Sycophants
This commentary from One Useful Thing examines the relationship between AI personality design and sycophantic behavior in large language models. The piece explores how model personality traits influence persuasion dynamics and user susceptibility to AI-generated agreement. It draws lessons from sycophancy research to understand broader risks in how AI systems are tuned to be agreeable.
Probe Trajectories Reveal Reasoning Dynamics in Large Reasoning Models
This paper investigates whether hidden representations of Large Reasoning Models (LRMs) can predict future model behavior by analyzing probe trajectories—the continuous evolution of concept probabilities across Chain-of-Thought reasoning tokens. The authors find that temporal trajectory features (volatility, trend, steady-state) significantly outperform single static probes, with max-pooling achieving up to 95% AUROC across safety and mathematics domains. Two methodological insights are offered: template-based training data matches dynamically generated responses in quality, and pooling strategy is critical to probe performance. The work positions probe trajectories as a complementary safety monitoring framework for LRMs where CoT faithfulness cannot be assumed.