Multimodal AI in 2025: seeing, hearing, and generating the world

What multimodal actually means

A multimodal model processes more than one type of input or produces more than one type of output. The most common configuration in 2025 was vision-language: a model that takes both images and text as input and produces text. GPT-4V (the "V" for vision) was an early example. By 2025, vision-language models were standard and unremarkable.

The interesting developments were in expanding to audio, video input, and video generation. Models that could listen to audio in real time, respond vocally, and maintain coherent multi-turn conversations without text intermediaries were shipping. Models that could watch a video and answer questions about specific moments in it were available. Models that could generate video from text with synchronised audio (Veo 3, from Google) were live.

The modality matrix

Real-time audio and why it matters

OpenAI's Advanced Voice Mode in GPT-4o removed the classic voice assistant architecture: speech-to-text, then LLM, then text-to-speech. Instead, the model processes audio end-to-end, which means it can hear tone and emotion, interrupt naturally, and respond with appropriate prosody. The result feels different from earlier voice interfaces. The latency is lower and the interaction feels more like talking to a person than issuing commands to a system.

Practical applications that opened up include real-time language tutoring with pronunciation feedback, live interpretation, customer service at scale with natural-sounding agents, and accessibility tools for users who find typing difficult. The risks are equally significant: voice cloning attacks, non-consensual impersonation, and the difficulty of distinguishing a real person from an AI agent in a phone call.

Video understanding: what the models see

Gemini 1.5 Pro with its two-million-token context window could ingest an entire feature-length film and answer questions about specific scenes, characters, and plot points. This was a research demonstration in early 2025. By October, developers were using it in practical applications: video transcription with speaker identification, automated content moderation, sports analysis, and surveillance summarisation.

The accuracy was not perfect. Models would miss actions that happened quickly or in the background, hallucinate events that did not occur, and struggle with overlapping conversations in audio. But for tasks like "summarise this four-hour meeting recording and identify the five most important decisions," the tools worked well enough to be genuinely useful.

The combination of video understanding and video generation creates a loop: AI can now watch a film, understand its style, and generate new footage in that style. The implications for creative production, advertising, and education are significant. The implications for synthetic media and misinformation are equally so.

The architecture behind it

Most modern multimodal models use a shared representation space. Visual and audio inputs are converted to token-like representations using encoders trained jointly with the language model. The language model then reasons over a combined sequence of text tokens and visual/audio tokens. This approach means the model can apply the same reasoning capabilities it developed for text to inputs in other modalities.

The training challenge is alignment: ensuring that visual representations carry the same conceptual content as their textual descriptions. Contrastive learning, where the model learns to align image and text representations of the same concept, was the key technique that made this work at scale. CLIP (from OpenAI) was the foundational work; everything since has built on variations of that architecture.

What comes next

The obvious next step is more seamless integration across modalities during generation. Current systems are good at understanding multiple modalities and generating text, and increasingly good at generating individual modalities like images or video. The frontier is generating coherent combinations, like a document that includes auto-generated charts grounded in the numbers in the text, or a presentation where the visuals are generated to match the spoken content.

The other frontier is grounding in the physical world. Models that can process sensor data, understand spatial relationships, and control physical systems are the connective tissue between AI and robotics. The same architecture that makes a model good at watching a video also makes it better at understanding what a camera-equipped robot is seeing. October 2025 was a moment when those two curves, software AI and physical AI, were visibly converging.