5arXiv cs.LG (Machine Learning)·
Impossibility result: spectral predictability indices cannot predict when context, retrieval, or foundation models help in time-series forecasting
A new arXiv preprint proves an impossibility result showing that power-spectrum-based predictability indices cannot determine whether adding context (longer lookback, retrieval, or pretrained models) will improve time-series forecasting, because such indices are invariant under phase randomization while the value of retrieval and foundation models is not. The authors introduce a label-free diagnostic called the 'coverage deficit' that measures beyond-spectrum structure as the gain of analog over linear prediction. Experiments across seven benchmarks validate the distinction: retrieval value collapses under spectrum-preserving surrogates while spectral indices remain frozen, and the new diagnostic predicts the sign of beyond-spectrum value where spectral indices fail. No new forecaster is introduced; the contribution is the theoretical distinction, controlled comparisons, and a deployment diagnostic.
Related guides (2)
Related events (8)
SPECTRA: Synthetic IR Test Collections with Relevance Oracles and Controlled Distractor Diagnostics
SPECTRA is a reproducible framework for generating synthetic information retrieval test collections, separating latent topical structure, surface text realization, and query intent generation to produce deterministic relevance oracles without human annotation. A Python prototype generated corpora up to 60,000 documents at roughly 12K–14K documents per second, with graded relevance labels for 96 queries. Controlled distractor experiments showed BM25 nDCG@10 degrading from 1.00 at 2% distractors to 0.43 at 36%, demonstrating the framework's utility for exposing retrieval system failure modes before expensive real-world collection construction. The authors position SPECTRA as a diagnostic complement to Cranfield/TREC-style evaluation rather than a replacement for human judgment.
Spectral initialization schemes for LM pretraining show diagnostic value but no performance gain
Researchers analyze weight spectra across eleven pretrained GPT-2-style checkpoints varying in size, language, and training corpus, finding consistent depth-wise patterns in Frobenius norm and effective-rank entropy. They construct initialization schemes that mimic these spectral profiles and compare them against standard initialization methods. Despite visibly altering structural spectral patterns, the proposed initializers do not yield performance improvements over pretrained-weight reuse. The results suggest pretrained spectra are useful diagnostics of model structure but that coarse spectral matching is insufficient for effective transfer.
Zero-shot LLMs fail to beat baselines on stock prediction; explainability signals retain practical value
A new arXiv preprint evaluates zero-shot NLP pipelines for predicting short-term stock movements from financial news, finding that across multiple models and prediction horizons, zero-shot approaches consistently fail to outperform simple baselines, with especially weak performance on negative price movements. The authors introduce a multi-layered explainability framework linking predictions to token-, article-, and aggregate-level evidence, finding that explainability signals can reliably distinguish trustworthy from unreliable predictions even when accuracy is low. The work argues for a shift toward decision-support systems emphasizing transparency and uncertainty awareness rather than raw predictive accuracy.
Predictability as a Fine-Grained Privacy Metric Complementary to Differential Privacy
A new arXiv preprint introduces 'privacy via predictability,' a framework that measures privacy leakage as the incremental gain in an attacker's ability to predict sensitive information after observing an algorithm's output, conditioned on the attacker's prior knowledge. The authors show predictability and differential privacy are generally incomparable, but that predictability implies mutual-information DP in worst-case regimes. They develop a generalized method of moments framework for asymptotic analysis and derive a predictability-calibrated output perturbation scheme for empirical risk minimization. The work positions predictability as a complementary, finer-grained alternative to DP for settings where attacker knowledge and query families can be specified.
Aggregate accuracy masks per-example prediction instability from task-irrelevant context in LLMs
A new arXiv paper demonstrates that state-of-the-art LLMs appear robust to task-irrelevant context at the aggregate level, but this stability conceals significant per-example prediction flips. Even semantically meaningless pseudo-words prepended to benchmark questions can shift model predictions on a subset of examples, with gains and losses roughly canceling out in aggregate. The instability is modulated by context type, length, test-time compute, and model development stage, and the affected examples are largely model-specific. The authors argue this reveals 'tail risks' hidden by standard aggregate accuracy metrics, motivating per-example reliability evaluation.
Study finds SAE unstable features reflect reproducible subspaces, not pure noise
A new arXiv paper investigates feature stability in sparse autoencoders (SAEs), measuring the probability that individual learned features reappear across independent training runs. The authors find a functional asymmetry: stable features carry most reconstruction-relevant signal, while unstable features are individually non-reproducible but concentrate in reproducible lower-rank subspaces, suggesting seed dependence reflects basis ambiguity rather than noise. A synthetic model confirms that low-rank ground-truth features can be recovered at the subspace level even when individual SAE latents are non-identifiable across seeds. The work has direct implications for interpretability research that relies on SAE features as meaningful, stable units of analysis.
WorldKernel: Formalizing world models as coupling kernels over counterfactual worlds
A new arXiv preprint identifies a structural failure mode in prediction-based world models: strong predictors can recover the diagonal of a counterfactual coupling kernel (ordinary posteriors) but systematically fail on off-diagonal cross-world couplings, collapsing to point estimates that are sometimes provably inadmissible. The authors formalize a world model as a positive semidefinite kernel K(T,T') over admissible possible worlds, showing the off-diagonal encodes counterfactual structure that more data cannot resolve. They demonstrate that PSD constraints provide partial identification bounds computable in polynomial time, that ontological axioms tighten these bounds, and that targeted constraint learning ('scars') closes the gap faster than untargeted approaches. The work has implications for causal reasoning in AI systems and the theoretical limits of learned world models.
Beyond Prediction Accuracy: Target-Space Recovery Profiles for Evaluating Model-Brain Alignment
This paper introduces a framework for evaluating alignment between artificial vision models and the human visual cortex that goes beyond scalar prediction accuracy. Using repeated fMRI data from the Natural Scenes Dataset, the authors decompose brain response spaces into reproducible dimensions and measure which of these dimensions are recovered by model predictions. A key finding is that pretrained and randomly initialized models can achieve similar prediction accuracy while showing distinct recovery profiles, revealing that accuracy alone can mask fundamental model-brain mismatches. The framework also enables brain-to-brain comparisons as a diagnostic human reference baseline.