- Modern frontier models like GPT-4o, Gemini 2.5, and Claude 3.5 Sonnet can process images, audio, and video alongside text in a single context.
- Models use separate vision encoders to convert images into embeddings or train natively on multimodal data from the start.
- Google's Gemini and Meta's Llama 4 train from scratch on interleaved text, image, audio, and video data for deeper integration.
- Analyzing medical imaging, reading handwritten notes, generating code from UI screenshots, and providing accessibility descriptions.
- Real-time voice assistants, video summarization, accessibility transcription, and customer call analysis for sentiment and topics.
- Multimodal models collapse specialized applications into a single flexible system that handles multiple data types simultaneously.
Beyond Text: Multiple Modalities
A multimodalMultimodalCapable of processing and generating multiple types of data — such as text, images, audio, and video — within a single model.Learn more → AI model can process and generate multiple types of data simultaneously. Early LLMs were purely text-in, text-out. Modern frontier models — GPTGPTGenerative Pre-trained Transformer — the model architecture and family name behind OpenAI's most famous models, from GPT-2 to GPT-5.Learn more →-4o, Gemini 2.5, Claude 3.5 Sonnet, Llama 4 Maverick — can process images, describe what they see, analyze charts and diagrams, and reason about visual and textual information together in a single context.
Modalities include: text (obviously), images (static photos, diagrams, screenshots), audio (speech, music), video (sequences of frames), and structured data. The most widely deployed multimodal capability today is vision — virtually all frontier models now accept images as input. Audio understanding and video analysis are rapidly maturing.
How Vision Is Added to LLMs
The dominant approach is to use a separate vision encoderEncoderThe transformer component that processes input sequences into contextual representations, forming the foundation of understanding-focused models like BERT.Learn more → (typically based on CLIP or a similar vision-language pre-training model) to convert images into embeddings that the LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → can process. The image is divided into patches, each patch is encoded into a vector, and these vectors are projected into the LLM's embeddingEmbeddingA numerical vector representation of text, code, or images that captures semantic meaning — similar items have similar vectors, enabling search, clustering, and retrieval.Learn more → space where they're treated alongside text tokens.
This approach is efficient because it reuses a strong pre-trained visual encoder. The LLM learns, during multimodal fine-tuningFine-tuningThe process of further training a pre-trained model on a smaller, task-specific dataset to specialize its behavior for a particular use case.Learn more →, to ground its language generation in visual features. Llava, a popular open-source multimodal framework, demonstrated this approach — a modest visual encoder combined with Llama produced competitive vision-language models.
Native Multimodal Training
Google's Gemini and Meta's Llama 4 take a different approach: native multimodal training from the start. Rather than adding vision as a post-hoc capability, these models are trained from scratch on interleaved text, image, audio, and video data. This produces more deeply integrated understanding — the model doesn't think of images as 'add-ons' to text processing.
GPT-4o ('omni') also pursues native multimodality, processing audio, vision, and text with the same model architecture. This enables capabilities like real-time voice conversation with visual awareness — features that would be awkward or impossible with bolted-on multimodal approaches.
What Multimodal AI Enables
Vision: analyzing medical imaging alongside reports, describing products from photos, reading handwritten notes, understanding whiteboard diagrams, debugging UI from screenshots, accessibility descriptions for visually impaired users. Code from screenshots: take a photo of a UI mockup and generate the HTML/CSS — a common developer workflow.
Audio and video: real-time voice assistants with conversational awareness, video summarization and search, accessibility transcription, analyzing customer call recordings for sentiment and topics. Each new modality unlocks applications that were impossible or required specialized models — multimodal models collapse these into a single flexible system.