What this area covers
Multimodal AI is the project of building systems that perceive and generate across more than one sensory channel — text, images, audio, video, and physical-world signals — ideally within a single coherent model rather than a pipeline of specialists. The thread spans from early contrastive vision-language pre-training through today's frontier of computer-use agents, synchronized video-audio generation, and embodied robotics.
Why it matters
Language models alone can reason about the world only as far as text describes it. Multimodal capability closes the gap between AI and the actual richness of human-generated information: most documents contain images, most interfaces are visual, most real-world tasks involve perception beyond reading. The practical payoff is direct: models that can see screens can operate software; models that can hear can transcribe and respond to voice; models that can generate video can prototype creative and simulation workflows that previously required specialist pipelines.
Phase 1 — Specialist foundations (2021–2023)
The modern multimodal era begins with CLIP (January 2021), OpenAI's contrastive vision-language model trained on image-text pairs from the web. CLIP demonstrated that a model could learn visual concepts from natural language supervision alone, enabling zero-shot image classification — the same transfer-learning insight that GPT-2 and GPT-3 had shown for text. It became the backbone of nearly every subsequent vision-language system.
Whisper (September 2022) did the analogous thing for speech: an open-weights ASR model trained on 680,000 hours of multilingual web audio, approaching human-level English transcription and handling accents, noise, and technical vocabulary robustly. Whisper established the open baseline that all subsequent speech models are measured against.
GPT-4 (March 2023) brought multimodal inputs to a frontier-scale language model for the first time, accepting images alongside text. By November 2023, GPT-4 Turbo with Vision was available via API, and ChatGPT had gained voice input and speech output — multimodal interaction moved from research demo to consumer product.
Phase 2 — Unified architectures and video (2024)
The pivotal architectural shift arrived with GPT-4o (May 2024). Unlike prior systems that routed modalities through separate models, GPT-4o processed audio, vision, and text natively in a single architecture — "Omni" in name and in mechanism. OpenAI positioned it as their primary production model going forward, and the announcement included a live demo of real-time voice conversation with emotional responsiveness. This set the template that subsequent flagship models would follow.
Simultaneously, the video generation track matured. Sora's research preview (February 2024) introduced a transformer operating on spacetime patches of video latent codes, trained jointly on videos and images of variable durations and resolutions, capable of generating up to one minute of high-fidelity video. OpenAI explicitly framed scaling video generation as a path toward physical world simulation — a claim that would echo through subsequent work on world models. Sora launched publicly in December 2024 at up to 1080p and 20 seconds.
On the open-weights side, Llama 3.2 (September 2024) added vision to Meta's Llama family for the first time, alongside lightweight 1B and 3B variants targeting edge and mobile deployment. Qwen2.5-VL (January 2025) from Alibaba extended the open vision-language frontier with 3B/7B/72B variants. The open ecosystem was catching up to closed models on vision understanding.
Claude 3 (March 2024) introduced multimodal vision capabilities to Anthropic's model family, with near-perfect recall on long-context evaluations alongside image understanding — marking Anthropic's entry into the vision-language tier.
Phase 3 — Computer use, audio generation, and embodiment (2025)
Three distinct multimodal frontiers opened in 2025, each representing a qualitatively different kind of perception-action loop.
Computer use emerged as a distinct capability class. Anthropic's Claude 3.5 Sonnet computer use beta (August 2025) enabled the model to control a computer by interpreting screenshots and issuing pixel-level cursor and keyboard commands, scoring 14.9% on the OSWorld benchmark — roughly double the next-best AI at the time, though well below the human-level range of 70–75%. OpenAI followed with its Computer-Using Agent (January 2025), combining GPT-4o's vision with reinforcement learning for GUI navigation. Google DeepMind released a Gemini 2.5 Computer Use preview (October 2025) via API. The race to close the gap with human-level GUI operation was on.
Video generation advanced on two fronts. Sora 2 (September 2025) added synchronized dialogue and sound effects — the first major video model to generate audio alongside video — alongside improved physics simulation and steerability. Google DeepMind's Veo 3 and Imagen 4 (May 2025) extended the competitive landscape, with Veo 3 targeting professional media production alongside a filmmaking tool called Flow.
Embodied AI became a serious product category. Google DeepMind's Gemini Robotics (March 2025) applied Gemini's multimodal capabilities to robotic systems that perceive, plan, and act in physical environments. Gemini Robotics 1.5 (October 2025) extended this to multi-step task execution and tool use. Gemini Robotics On-Device (June 2025) pushed the model to run locally on robotic hardware, targeting fast task adaptation without cloud inference. Genie 3 (October 2025) demonstrated real-time interactive 3D world generation at 24fps and 720p — not a robot, but a generative world model that responds to user actions, framing the longer-term goal of simulation-grade world understanding.
Speech also saw open-weights progress: Mistral's Voxtral (July 2025) released 24B and 3B open-weight speech understanding models under Apache 2.0, supporting long-form audio up to 30–40 minutes, multilingual transcription, and function-calling from voice — benchmarking competitively with GPT-4o mini and Gemini 2.5 Flash on audio understanding.
Phase 4 — Unification and the open-weights convergence (2025–2026)
The current phase is characterized by two simultaneous pressures: closed frontier models converging on full omnimodality, and open-weights models rapidly closing the gap on individual modalities.
Anthropic's acquisition of Vercept (February 2026) — a team specializing in AI perception for computer use, co-founded by Kiana Ehsani, Luca Weihs, and Ross Girshick — was a direct investment in closing the computer-use perception gap. The payoff was immediate: Claude Sonnet 4.6 (March 2026) reached 72.5% on OSWorld, approaching human-level performance on tasks like navigating spreadsheets and completing web forms, up from under 15% in late 2024. Claude Opus 4.7 (May 2026) added higher image resolution and improved vision capabilities alongside new cybersecurity safeguards.
On the open side, Mistral Small 4 (March 2026) unified capabilities previously split across three separate Mistral models — Magistral (reasoning), Pixtral (multimodal), and Devstral (coding agents) — into a single 119B MoE model with native text and image input, released under Apache 2.0. This is the clearest example yet of open-weights multimodal unification: one model, one license, all modalities.
Meta's Muse Spark (April 2026), the debut product of Meta Superintelligence Labs, is natively multimodal with visual chain-of-thought and multi-agent orchestration — and notably closed-weights, marking a strategic departure from Meta's open Llama strategy. Google DeepMind announced Gemini Omni (May 2026), signaling a unified-modality extension of the Gemini line, and Gemini 3.5 (May 2026) emphasized agentic capabilities built on multimodal foundations.
The central tension: unified vs. specialized
The events in this bundle trace a persistent architectural debate. The unified-architecture camp (GPT-4o, Gemini 3+, Gemini Omni, Muse Spark) argues that a single model trained end-to-end across modalities develops richer cross-modal representations and enables tighter integration — a model that truly "thinks" in images and audio, not one that translates them into text first. The specialized camp argues that per-modality models (Whisper for ASR, Voxtral for speech, Veo 3 for video) can be optimized more deeply for their domain and composed flexibly.
In practice, the frontier is moving toward unified architectures for perception and reasoning, while generation (especially video and audio) remains partially specialized — Sora 2 and Veo 3 are not the same models as GPT-5.x and Gemini 3.x. Whether generation fully merges with understanding in a single model is the open question this thread will track.
Where it's heading
The events point toward three active frontiers:
1. Computer use reaching human-level GUI performance. The OSWorld trajectory (14.9% → 72.5% in roughly six months) suggests this benchmark will be saturated; the next challenge is reliability and safety in open-ended real-world environments.
2. Video + audio generation as a unified capability. Sora 2's synchronized audio is the first step; the gap between video generation and interactive world simulation (Genie 3) is the longer-term target.
3. Embodied multimodal AI. Gemini Robotics On-Device represents the push from screen-level perception into physical-world action — the most demanding test of whether unified multimodal representations actually generalize to the real world.
