5arXiv cs.AI (Artificial Intelligence)·
MDTransformer: Mode-division photonic transformer accelerator with inverse-designed coherent crossbar
Researchers propose MDTransformer, a hardware-software co-design for photonic transformer acceleration using mode-division optical dataflow instead of multi-wavelength approaches. The system uses inverse-designed multi-mode couplers and Mach-Zehnder IQ modulators to form a compact photonic tensor core, with each guided mode acting as an independent computational lane for four-fold parallelism per waveguide. Evaluated against DeiT and BERT workloads, MDTransformer achieves 40.4% area reduction, 63.6% power saving, and 40.6% energy saving over prior photonic transformer accelerators while maintaining comparable latency.
Related guides (2)
Related events (8)
OrpQuant: Geometric Orthogonal Residual Projection for Multiplier-Free Power-of-Two Transformer Quantization
This paper introduces Orthogonal Residual Projection (ORP), an algorithm-hardware co-design framework for ultra-low-bit quantization of LLMs and Vision Transformers targeting edge deployment. ORP addresses the structural limitations of Power-of-Two (PoT) quantization by formulating quantization as a dual-basis geometric projection that synthesizes higher-resolution residual lattices using only shift-and-add operations, eliminating multipliers. At 3-bit (W3/A16), ORP achieves 6.10 perplexity on LLaMA-2-7B, competitive with MAC-intensive baselines like AWQ, while reducing full-model calibration time to ~15 minutes. RTL synthesis at 28nm confirms hardware efficiency by mitigating timing bottlenecks from dense multiplier trees.
Introducing Optimum: The Optimization Toolkit for Transformers at Scale
Hugging Face announced Optimum, an optimization toolkit designed to accelerate Transformers models on various hardware backends. The toolkit aims to bridge the gap between Transformers model development and hardware-specific optimizations from partners. It provides a unified interface for quantization, pruning, and hardware-accelerated inference across different accelerators.
Memory-efficient Diffusion Transformers with Quanto and Diffusers
This Hugging Face blog post describes integrating the Quanto quantization library with the Diffusers framework to reduce memory requirements for diffusion transformer models. The approach enables running large image/video generation models on consumer-grade hardware by applying int8 and int4 quantization to model weights. The post covers practical implementation details and benchmarks showing memory savings for models like Flux and others in the diffusion transformer family.
Loopie: Looped MoE Transformer achieves gold-medal performance at IMO and IPhO
Researchers introduce Loopie, a looped Transformer series using Mixture-of-Experts architecture in two sizes (20B/2B active and 6B/0.6B active parameters). The work directly addresses the longstanding challenge that scaling parameters outperforms looping under equal compute budgets, demonstrating that Loopie substantially outperforms vanilla Transformer baselines at the same compute. A novel post-training pipeline yields strong reasoning capabilities, with Loopie achieving gold-medal performance at the 2025 IMO and IPhO competitions without tool use.
Roofline-inspired scaling model predicts Transformer fine-tuning energy consumption across GPU configurations
A new arXiv preprint presents a framework for modeling energy consumption during Transformer training on multiple GPUs, using BERT architectural sweeps to relate measured energy to proxies for compute, memory traffic, and hardware efficiency. The approach adapts roofline modeling with a speedup-based hardware-efficiency factor that accounts for tensor parallelism and fully sharded data parallelism. The resulting scaling law accurately predicts training energy across heterogeneous configurations, targeting sustainable and cost-aware system design.
MODUS: Decoder-only architecture for any-to-any multimodal modeling without modality-specific heads
Researchers at EPFL introduce Modus, a decoder-only model that treats all modalities symmetrically for any-to-any prediction without modality-specific heads, losses, or task pipelines. Unlike prior any-to-any models that use encoder-decoder or diffusion architectures trained from scratch, Modus leverages pre-trained decoder-only models as a prior. The model supports chained generation through intermediate modalities and cross-modal self-verification, achieving competitive performance with specialist and multitask baselines across benchmarks. All materials are open-sourced.
Variable-Width Transformers: X-shaped architecture outperforms uniform-width baselines with 22% fewer FLOPs
Researchers propose the ><former (X-shaped transformer), a decoder-only architecture that uses wider early and late layers with narrower middle layers, implemented via a parameter-free residual resizing mechanism. Evaluated on models from 200M to 2B dense parameters and 3B MoE, the architecture consistently outperforms parameter-matched uniform-width baselines on language modeling loss. The design yields a 22% reduction in FLOPs and 15% reduction in KV cache memory under fitted scaling curves, suggesting nonuniform width allocation is a viable path to more compute-efficient language models.
HAMON: Passive diffractive optical system for long-horizon time-series forecasting
HAMON is a proposed forecasting architecture that replaces learned digital sequence-mixing layers with a passive diffractive optical core: historical values are encoded onto an optical aperture and cascaded trainable phase masks with free-space diffraction produce forecasts in the output field. At inference, prediction requires only a single passive optical propagation pass with no digital temporal mixing. The system outperforms strong digital baselines on ETTm2 and ETTh2 benchmarks by up to 14% MSE improvement, though it trails on high-channel-count datasets like Traffic and Electricity. The work raises a substrate-level question about whether forecasting operators need to be implemented digitally at all, and defines a concrete target for optical computing hardware.