How to build a multimodal RAG system for documents with images

Traditional RAG ignores charts, diagrams, and images embedded in documents. Multimodal RAG retrieves and reasons over both text and visual content. This guide covers document parsing, image embedding, and hybrid retrieval.

Why standard RAG misses half the content

Many business documents — annual reports, research papers, technical manuals — contain charts, tables, and diagrams that convey critical information not repeated in the text. Standard RAGRAGRetrieval-Augmented Generation — a technique that enhances LLM responses by first retrieving relevant documents from an external knowledge base and including them in the prompt.Learn more → pipelines that only process text will miss this information entirely, leading to incomplete or incorrect answers.

MultimodalMultimodalCapable of processing and generating multiple types of data — such as text, images, audio, and video — within a single model.Learn more → RAG addresses this by extracting images from documents, generating text descriptions or embeddings of those images, and including them in the retrieval index. When a query requires information from a chart, the system can retrieve and reason over the image directly.

Extract images from PDFs

Use `pymupdf` (formerly PyMuPDF) to extract both text and images from PDFs: `import pymupdf; doc = pymupdf.open('report.pdf'); for page in doc: text = page.get_text(); image_list = page.get_images()`. For each image, render it as a PNG: `img_data = doc.extract_image(xref)['image']`.

Alternatively, render each page as a high-resolution image and use a vision model to extract all text and describe all figures — this avoids separate text extraction and catches text embedded in images (scanned documents). Use 200+ DPI for legible text: `pix = page.get_pixmap(dpi=200); pix.save('page.png')`.

Generate descriptions for images

For each extracted image, send it to GPTGPTGenerative Pre-trained Transformer — the model architecture and family name behind OpenAI's most famous models, from GPT-2 to GPT-5.Learn more →-4o or Gemini with a specific description promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more →: 'Describe this chart/diagram in detail. Include: (1) the chart type, (2) what is being measured, (3) all data values, (4) trends or patterns, (5) any labels or annotations. Be comprehensive — your description will be the only representation of this image in a search index.'

Store the description alongside the image and associate it with the page number and document metadata. The description is what gets embedded and indexed. At retrieval time, you can include the image itself in the prompt for visual groundingGroundingThe process of connecting an LLM's outputs to verified external information sources to improve accuracy and reduce hallucinations.Learn more →.

Index and retrieve

Create two separate collections in your vector databaseVector DatabaseA database optimized for storing and searching embedding vectors, enabling fast semantic similarity search across millions of text chunks or documents.Learn more →: one for text chunks, one for image descriptions. Embed both with the same text embedding modelEmbedding ModelA specialized neural network that converts text into high-dimensional vectors, enabling semantic search, clustering, and retrieval applications.Learn more →. At query time, search both collections and merge the results by relevance score.

For the final generation step, include the retrieved images directly in the prompt: retrieve the top-3 text chunks and top-2 image descriptions, then build a multimodal prompt with both the text and the actual images. The vision model can then reason over the visual content directly rather than relying on the text description.