Entity · model
Pythia
modelactive
pythia-6884907e·5 events·first seen Aliases: Pythia
Co-occurring entities
A Multi-Agent System for Autonomous, Fine-Tuning-Free Clinical Symptom Detection: Development and Validation StudyBERTNatural Ungrokking: Asymmetric Control of Which Rules Survive PretrainingMany Circuits, One Mechanism: Input Variation and Evaluation Granularity in Circuit DiscoveryShannon-Hartley TheoremShannon Scaling Lawquantization-induced degradationcatastrophic overtrainingsignal-to-noise ratio (SNR)OLMo2WikiText-2layer pruningQwen3-4Bswap-KLLlama-3.1-8B
More like this (12)
Recent events (5)
Pythia: Multi-agent system for fine-tuning-free clinical symptom extraction from notes
Researchers present Pythia, a multi-agent system that autonomously writes and optimizes extraction prompts for clinical concepts in medical notes without manual prompt engineering or model fine-tuning. Running on locally hosted open-weights models, Pythia achieved mean sensitivity of 0.76 and specificity of 0.95 across 72 symptoms from 400 clinical notes, outperforming a curated lexicon on specificity and a per-concept BERT classifier on both metrics. The system's key advantage is recovering high specificity (0.97) for concepts where lexicon-based approaches over-trigger, while remaining deployable on local infrastructure for data privacy. Sensitivity degrades below 5% prevalence, a noted limitation for rare findings.
Natural Ungrokking: Pretraining Can Silently Erase Learned Rules Without Loss Signal
A new arXiv preprint documents a phenomenon called 'natural ungrokking,' in which small language models learn a generalizable rule mid-pretraining (e.g., pronoun-gender agreement) and then lose it entirely by later steps, with no trace in the loss curve. The key predictor of rule survival is corpus support frequency — how often the training stream shows the rule winning over competing surface patterns. Critically, the forgetting is asymmetric: targeted data edits can destroy a rule on demand, but injecting up to 450x the sustaining support level cannot restore it. The findings are validated on public Pythia checkpoints and were pre-registered before data collection.
Phantom specialization in circuit discovery: structural differences don't imply distinct mechanisms
A new arXiv preprint challenges a core assumption in mechanistic interpretability: that structurally different circuits discovered for the same task imply distinct computational mechanisms. Using Literal Sequence Copying across token-frequency bands in five Pythia models (70M–1.4B), the authors extract 75 circuits and show that structurally distinct circuits implement the same computation, with band-specific edges transferring broadly and a shared core recovering ≥99% of circuit performance. The paper introduces the term 'phantom specialization' for this pattern and argues that standard source-level evaluation inflates apparent faithfulness, while edge-level evaluation and cross-condition transfer tests are needed to detect the many-to-one mapping from structure to function.
Shannon Scaling Law: A Noisy-Channel Framework for LLM Capacity and Non-Monotonic Training Phenomena
Researchers propose the Shannon Scaling Law, a theoretical framework that models LLM training as information transmission over a noisy channel using the Shannon-Hartley theorem. By mapping model parameters to channel bandwidth and training tokens to signal power, the framework introduces a fundamental SNR-based capacity limit that explains non-monotonic phenomena like catastrophic overtraining and quantization-induced degradation that classical power-law scaling laws cannot capture. Validated on Pythia and OLMo2 under Gaussian noise, quantization, and fine-tuning perturbations, the law achieves strong R² scores and successfully extrapolates from 6.9B to 12B parameter models trained on up to 307B tokens. The framework outperforms both classical and perturbation-aware scaling laws, predicting U-shaped performance degradation when SNR is insufficient.
Layer Equivalence Is Not a Property of Layers Alone: How You Test Redundancy Changes What You Find
This paper distinguishes two protocols for measuring transformer layer redundancy—replacement (can one layer substitute for another in place?) and interchange (do two layers approximately commute when swapped?)—and shows they can disagree substantially. Experiments on Pythia (410M, 1.4B) and 8B-scale models (Qwen3-8B, Llama-3.1-8B) reveal that the protocol gap grows during training and can change which layers appear safe to prune by several-fold. Notably, Qwen3-8B shows interchange-guided removal is far safer than replacement-guided at the same layer budgets, while Llama-3.1-8B ties the two protocols despite lower interchange KL. The authors recommend scoring both swap-KL metrics before any layer removal or merging, requiring only unlabeled forward passes.