4arXiv cs.AI (Artificial Intelligence)·
Graph Convolutional Attention: Spectral perspective improves graph denoising and diffusion
A new arXiv preprint introduces Spectral Attention and its practical realization, Graph Convolutional Attention (GCA), motivated by a theoretical analysis showing that linear attention is suboptimal for graph denoising tasks. The authors prove that standard linear attention can only learn an average spectral filter over the training distribution, while GCA provably outperforms it by a margin tied to spectral diversity. Empirically, GCA improves graph denoising and diffusion on synthetic and real datasets, and in the DiGress graph diffusion model it matches standard graph-transformer performance without expensive structural feature computation, enabling faster inference when combined with PEARL positional encodings.
Related guides (1)
Related events (8)
AGDO: Attention-guided denoising and optimization framework improves diffusion language model reasoning
Researchers propose AGDO, a framework that replaces random masking in diffusion large language models (dLLMs) with attention-guided denoising order and token weighting during fine-tuning and reinforcement learning. The work is motivated by an empirical finding that tokens with stronger attention to unmasked context are more stable and critical for reasoning. Experiments on math and coding benchmarks show AGDO outperforms existing post-training methods for dLLMs, advancing the case for attention-aware training in parallel-decoding language models.
Functional Attention: Reinterpreting Attention as Functional Correspondences for Operator Learning
This paper introduces Functional Attention, a novel attention mechanism for operator learning that replaces standard softmax token-wise affinities with structured linear operators inspired by geometric functional maps. The approach treats attention as a correspondence between adaptive bases rather than discrete tokens, yielding a resolution-invariant, globally-aware representation. Experiments show competitive or state-of-the-art performance on PDE solving, 3D segmentation, and regression tasks, with robustness to varying discretizations.
P-K-GCN: Physics-augmented Koopman-enhanced Graph Convolutional Network for spatiotemporal super-resolution
Researchers propose P-K-GCN, a framework combining graph convolutional networks, Koopman operator theory, and physics-informed loss functions for spatiotemporal super-resolution on irregular geometries. The method linearizes nonlinear dynamics in a latent space and enforces physical constraints to improve reconstruction fidelity. Theoretical analysis claims guaranteed error reduction via Rademacher complexity bounds. The framework is evaluated on reconstructing high-resolution cardiac electrodynamics from sparse 3D heart geometry measurements.
PromptGNN-sim: Bidirectional GNN-LLM fusion framework for text-attributed graph learning
Researchers introduce PromptGNN-sim, a bidirectional structure-semantic fusion framework that jointly trains a Graph Attention Network and an LLM for text-attributed graph learning. The system uses GAT-based neighborhood selection to generate structure-aware prompts for the LLM, with cross-modal contrastive learning and cross-attention aligning both components during training. Evaluated on six datasets including Cora, Pubmed, and WikiCS, it outperforms classical GNNs, standalone LLMs, and prior GNN-LLM fusion methods on cross-task transfer, cross-dataset generalization, and sparse perturbation settings.
Lie-Algebra Attention: tokens as bare matrix Lie group elements with closed-form geometric scores
A new arXiv preprint introduces Lie-Algebra Attention, an attention mechanism where tokens are elements of a matrix Lie group rather than feature vectors, with pairwise attention scores computed as the closed-form algebra norm of the relative pose (log of the group inverse product). The construction achieves equivariance tautologically without representation-theoretic machinery such as irreps, spherical harmonics, or Clebsch-Gordan products, and extends to non-compact affine groups that existing methods cannot handle. Experiments on SE(2), SO(3), and Aff(2) sequence-completion tasks show the closed-form score matches or outperforms learned MLP kernels while using 50–80x fewer score parameters.
CARVE: Content-aware gating for linear attention recurrent models improves efficiency and quality over GDN-2
CARVE (Content-Aware Recurrent with Value Efficiency) is a new linear attention architecture that addresses three coupled defects in the GDN-2 delta-rule architecture by restricting erasure to the key axis rather than the value axis. This design choice is proven necessary and sufficient to enable the WY-form triangular chunk solver, enabling competitive training throughput with Transformers. At 1.3B parameters trained on 100B tokens, CARVE achieves lower perplexity than GDN-2, leads recurrent baselines on nine commonsense reasoning benchmarks, and sets state-of-the-art on RULER retrieval probes, while using 13% less peak memory and 19% fewer parameters at 0.4% throughput overhead.
Neuronal Stochastic Attention Circuit (NSAC) for Probabilistic Representation Learning
Researchers introduce NSAC, a biologically-inspired continuous-time attention architecture that models attention logits as solutions to an Ornstein-Uhlenbeck stochastic differential equation, drawing on C. elegans Neuronal Circuit Policy wiring to induce Gaussian distributions over attention weights. The architecture enables joint quantification of aleatoric and epistemic uncertainty via a two-term objective combining Gaussian negative log-likelihood with an epistemic-separation regularizer. Empirical evaluation spans irregular time-series function approximation, multivariate regression, long-range forecasting, Industry 4.0 tasks, and autonomous vehicle lane-keeping, showing competitive accuracy with well-calibrated uncertainty estimates.
Cross-attention attribution reveals how natural language instructions shape speech diffusion model outputs
Researchers adapt the DAAM cross-attention attribution framework to speech diffusion models for the first time, applying it to CapSpeech-TTS to analyze how individual caption tokens influence acoustic output. The study analyzes 3,600 style-caption/transcript combinations across 25 layers and 24 ODE steps, producing per-token heatmaps. Key findings include that style tokens exhibit lower temporal variance than content tokens, style attention correlates with F0 and energy, and style conditioning peaks in early diffusion steps and deep layers. This is the first interpretability study of natural language conditioning in speech diffusion models.