What multimodal AI is
For most of AI's history, models dealt with one thing at a time: text in, text out. "Multimodal" AI breaks that constraint. A multimodal model can take in a mix of inputs — a photo, a spoken question, a video clip — and respond in kind. The goal, still being chased, is an AI that perceives the world the way people do: through sight, sound, and language all at once.
Why it matters
Most real-world tasks aren't text-only. A doctor reviewing a scan, a developer debugging a UI, a filmmaker editing footage — all of them work with images, audio, and video alongside words. AI that can only read text is a limited assistant for these jobs. AI that can see, hear, and act opens up a much wider range of genuinely useful applications.
Phase 1: Teaching AI to see (2021–2023)
The modern multimodal era traces back to CLIP (January 2021), a model from OpenAI that learned to connect images and natural-language descriptions. Instead of being trained to recognize specific categories, CLIP could match a photo to a sentence it had never seen before — a trick called zero-shot transfer. This became the backbone of nearly every vision-language system that followed.
GPT-4 (March 2023) brought image understanding into a flagship general-purpose model for the first time, accepting photos alongside text and producing human-level responses on professional benchmarks. By September 2023, ChatGPT had gained vision, voice input, and spoken responses — putting multimodal AI in front of millions of everyday users for the first time.
On the audio side, Whisper (September 2022) gave the open-source community a robust speech-recognition system trained on 680,000 hours of audio, handling accents, background noise, and dozens of languages with near-human accuracy.
Phase 2: Unification — one model for everything (2024)
The big architectural shift came in 2024. Rather than bolting separate vision or audio modules onto a text model, labs began building models where all modalities were native from the start.
GPT-4o ("Omni," May 2024) was the clearest statement of this direction: a single model that reasons across audio, vision, and text in real time, with no separate pipeline stages. OpenAI positioned it as their primary production model going forward.
Video generation arrived in a big way too. OpenAI's Sora (research preview February 2024; public launch December 2024) could generate up to a minute of high-fidelity video from a text description, running on a transformer architecture that treats video as patches of space and time — the same basic idea as a language model, just applied to moving images. OpenAI framed scaling video generation as a path toward AI that understands how the physical world works.
The open-weights world kept pace. Meta's Llama 3.2 (September 2024) added vision to the Llama family for the first time, and Alibaba's Qwen2.5-VL (January 2025) brought strong vision-language models to the open ecosystem in sizes from 3B to 72B parameters.
Phase 3: From perception to action (2025–2026)
Seeing and hearing wasn't the end goal — acting was. The most striking development of 2025–2026 has been AI that can operate software and physical robots.
Computer use — where an AI reads your screen like a human would and moves the mouse or types on your behalf — went from a research curiosity to a rapidly improving capability. Anthropic launched it in August 2025 with Claude 3.5 Sonnet scoring 14.9% on the OSWorld benchmark (roughly double the next-best AI at the time, though well below the human score of 70–75%). By early 2026, after Anthropic acquired the perception-specialist team Vercept, Claude Sonnet 4.6 had reached 72.5% on OSWorld — near human-level performance on tasks like navigating spreadsheets and filling out web forms. Google DeepMind launched its own Gemini 2.5 Computer Use model in October 2025, and OpenAI announced its Computer-Using Agent (CUA) in January 2025.
Video generation matured in parallel. Sora 2 (September 2025) added synchronized dialogue and sound effects — closing the gap between silent video and the full audiovisual experience. Google DeepMind released Veo 3 and Imagen 4 in May 2025 for video and image generation respectively, targeting creative and professional media production.
The most ambitious extension has been into the physical world. Google DeepMind's Gemini Robotics (March 2025) and Gemini Robotics 1.5 (October 2025) brought multimodal perception and reasoning into robotic systems that can act in physical environments. A companion Gemini Robotics On-Device model (June 2025) runs locally on robotic hardware without a cloud connection. DeepMind's Genie 3 (October 2025) went further still, generating interactive, navigable 3D environments in real time at 24fps and 720p — a demonstration of AI as a world-builder, not just a world-observer.
The open-weights multimodal ecosystem
Multimodal capability is no longer exclusive to the biggest closed labs. Mistral AI released Voxtral (July 2025), a family of open-weight speech-understanding models that outperform Whisper large-v3 and handle up to 30–40 minutes of audio with built-in Q&A and summarization. In March 2026, Mistral Small 4 unified vision, reasoning, and coding into a single 119B-parameter open-weights model — collapsing what had previously been three separate Mistral products into one.
Meta's Muse Spark (April 2026), from its newly formed Superintelligence Labs, introduced natively multimodal reasoning with visual chain-of-thought — though notably as a closed-weights model, a departure from Meta's usual open strategy.
Where things stand
The trajectory is clear: modalities that were once handled by separate specialist tools are converging into unified models, and those models are increasingly expected not just to perceive but to act. The open-weights ecosystem has closed much of the gap with closed labs on vision and speech. The remaining hard problems — consistent long-form video, reliable physical-world robotics, and computer use that works across arbitrary software — are all active areas of rapid progress.
For anyone building with AI today, the practical upshot is that "can it see?" and "can it hear?" are no longer differentiating questions. The new questions are: how well does it act, how reliably does it understand what it perceives, and how far can it go without a human checking in?
