4arXiv cs.LG (Machine Learning)·
GatedLinear: Adaptive routing of complementary linear bases for time series forecasting
A new arXiv preprint introduces GatedLinear, a lightweight forecasting framework that routes inputs across three specialized linear mechanisms — trend-seasonal, incremental drift, and phase-aligned recurrence — using a Tri-Factorized Fusion Gate. The gate disentangles routing decisions into channel-specific, horizon-aware, and phase-indexed components without heavy neural modules. Experiments on standard benchmarks claim state-of-the-art or competitive accuracy against larger foundational models with a substantially smaller parameter footprint and interpretable routing patterns.
Related guides (1)
Related events (8)
Feature steering via sparse autoencoders reduces look-ahead bias in LLM forecasting
Researchers apply sparse autoencoders to inspect LLM internal states during forecasting tasks, identifying features associated with time-aware versus look-ahead-biased reasoning. Amplifying time-awareness features causally reduces look-ahead bias while preserving general reasoning performance, whereas directly steering look-ahead-bias features has no effect. The work demonstrates that interpretable temporal features can shift LLMs toward more historically grounded forecasting. This is a mechanistic interpretability result with practical implications for LLM-based prediction systems.
Review Residuals: input-dependent gating for transformer residual connections shows scale-emergent gains
A new arXiv preprint introduces Review Residuals, a mechanism that replaces fixed-coefficient residual connections in transformers with a learned, input-dependent gate conditioned on both the current hidden state and the proposed sublayer update. Experiments across five model sizes (60M–1B parameters) find no advantage at small scale but statistically significant improvements over standard residuals and Highway-style gates at 590M and 1B parameters, with the benefit growing with scale. A key finding is that convex (Highway-style) gating reintroduces vanishing gradients beyond ~20 layers, while the additive identity-preserving form trains stably at all tested depths.
LLM Agent Framework for Last-Mile Time Series Forecasting Revision
This paper introduces a 'last-mile forecasting' framework where an LLM agent sits atop a statistical forecasting backbone to incorporate weakly structured business context—holidays, campaigns, expert feedback, external events—into decision-ready forecasts. The system uses tool-invocation for contextual retrieval, converts reasoning into explicit revision actions under safety constraints, and supports long-horizon forecasting via map-reduce decomposition with a memory bank for post-hoc reflection. The authors validate the approach through real-world case studies, positioning it as a bridge between statistical prediction and operationally usable forecasts.
Analysis-driven transformer linearization outperforms prior baselines on LLaMA and Qwen up to 32B
A new arXiv paper analyzes why post-hoc linearization of causal self-attention degrades model quality, identifying key-dependent rank-1 orthogonal projections as the mechanism softmax relies on and explaining why delta-style networks outperform gated accumulation. The authors introduce structural interventions—sink tokens, short convolutions, and fixed-budget cache routing—applied in a frozen-backbone regime. Scaling across LLaMA and Qwen models up to 32B parameters, the approach outperforms prior post-hoc linearization baselines on MMLU and matches long-context retrieval of adaptive-caching frameworks.
TiRex-2: xLSTM-based time series foundation model for multivariate and streaming forecasting
TiRex-2 is a recurrent xLSTM-based time series foundation model that extends the univariate TiRex to multivariate forecasting with past and future covariates, while supporting streaming inference at constant per-patch cost. The model uses a bidirectional time mixer and asymmetric grouped-attention variate mixer to handle future-known covariates without violating causality over target variables. A synthetic coupling pipeline enables scalable multivariate pretraining from univariate corpora. TiRex-2 claims state-of-the-art zero-shot performance on GIFT-Eval and fev-bench benchmarks with 38.4M–82.5M parameters depending on mode.
CARVE: Content-aware gating for linear attention recurrent models improves efficiency and quality over GDN-2
CARVE (Content-Aware Recurrent with Value Efficiency) is a new linear attention architecture that addresses three coupled defects in the GDN-2 delta-rule architecture by restricting erasure to the key axis rather than the value axis. This design choice is proven necessary and sufficient to enable the WY-form triangular chunk solver, enabling competitive training throughput with Transformers. At 1.3B parameters trained on 100B tokens, CARVE achieves lower perplexity than GDN-2, leads recurrent baselines on nine commonsense reasoning benchmarks, and sets state-of-the-art on RULER retrieval probes, while using 13% less peak memory and 19% fewer parameters at 0.4% throughput overhead.
Hindcast: A benchmark framework that closes data-leakage loopholes in LLM forecasting evaluation
A new arXiv preprint introduces Hindcast, an evaluation framework for LLM forecasters that addresses two systematic data-leakage channels in standard backtesting: post-event retrieval and training-data contamination from newer models. The system replays resolved Polymarket prediction markets against a frozen Reddit snapshot, grading models only on information available before a chosen cutoff date and comparing against contemporaneous market prices as a human-forecast baseline. Key finding: retrieval helps forecasting only when pre-event Reddit discussion existed; where only speculation was available, retrieval hurts performance. The framework is designed to remain valid as new models and markets emerge without going stale.
HAMON: Passive diffractive optical system for long-horizon time-series forecasting
HAMON is a proposed forecasting architecture that replaces learned digital sequence-mixing layers with a passive diffractive optical core: historical values are encoded onto an optical aperture and cascaded trainable phase masks with free-space diffraction produce forecasts in the output field. At inference, prediction requires only a single passive optical propagation pass with no digital temporal mixing. The system outperforms strong digital baselines on ETTm2 and ETTh2 benchmarks by up to 14% MSE improvement, though it trails on high-channel-count datasets like Traffic and Electricity. The work raises a substrate-level question about whether forecasting operators need to be implemented digitally at all, and defines a concrete target for optical computing hardware.