Multimodal RAG: GPU Planning for Document, Image and Video Retrieval
Overview
The gap between what a text RAG system indexes and what an enterprise actually knows is large. Scanned contracts, engineering drawings, presentation decks, screenshots in support tickets, recorded meetings and training videos hold a substantial share of institutional knowledge, and a text-only pipeline either skips them or extracts them badly. Multimodal RAG closes that gap by indexing visual and audio content directly. The infrastructure consequence is a sharp shift in where GPU time goes: ingestion becomes the dominant cost, and query serving stays comparatively cheap.
Key takeaways
- Ingestion dominates the GPU budget — processing an archive once is far larger than serving queries against it.
- Page-image embedding avoids extraction loss — treating a page as an image preserves layout, tables and diagrams.
- Video is the expensive modality — frame sampling strategy determines cost more than model choice.
- Audio is cheap and high-value — transcription unlocks meetings and calls at modest compute.
- Indian-language visual content is the weak spot, so validate on your own material before committing.
Two approaches to visual documents
The traditional path extracts text from a document — by OCR for scans, by parsing for digital files — and indexes that text. It works when documents are text-shaped and fails on everything else: a table becomes a jumble, a diagram becomes nothing, a chart’s meaning is lost entirely, and layout that carried semantics disappears.
The newer path embeds the page image itself using a vision-language embedding model, so retrieval matches a query against the visual document representation without an extraction step. Layout, tables and diagrams are preserved because nothing was thrown away. The cost is that embedding an image is more expensive than embedding text, and the index is larger. For document sets that are genuinely visual — engineering, finance, government forms — that cost is usually worth paying.
Where the GPU time goes
| Stage | Modality | Relative GPU cost | Frequency |
|---|---|---|---|
| Page image embedding | Documents | Moderate per page, large in aggregate | Once per document |
| Caption and description generation | Images, diagrams | High per item | Once per item |
| Transcription | Audio, video | Low relative to value | Once per recording |
| Keyframe extraction and embedding | Video | High; sampling-dependent | Once per video |
| Query embedding | All | Negligible | Per query |
| Generation with retrieved context | All | Moderate | Per query |
The asymmetry is the planning insight. Ingesting a corpus of a hundred thousand pages plus a few thousand hours of recordings is a substantial batch job measured in GPU-days. Serving queries against the result is a modest continuous load. That argues strongly for renting or sharing capacity for ingestion campaigns while owning capacity sized to serving — particularly since re-ingestion recurs whenever the embedding model changes.
Video is where budgets break
Video is not one modality but three: visual frames, audio, and any on-screen text. Processing every frame is unnecessary and prohibitively expensive, so the design decision is sampling. Uniform sampling at a low rate is simple but misses fast changes; scene-change detection samples adaptively and is usually the better default; and for screen recordings or lectures, sampling on slide transitions captures almost all the information at a fraction of the cost.
The practical guidance is to treat the audio track as the primary index for most enterprise video. Transcription is cheap relative to visual processing, and for meetings, training and support calls the spoken content carries most of the retrievable meaning, with sampled keyframes providing supporting visual context. Reversing that priority — heavy visual processing with audio as an afterthought — multiplies cost for little retrieval benefit.
Retrieval design across modalities
A common mistake is embedding everything into one vector space and hoping similarity works across modalities. In practice, results improve when each modality is indexed with an appropriate model and the fusion happens at the ranking stage rather than in the embedding space. Retrieve candidates per modality, then re-rank the merged set against the query with a cross-encoder that can compare heterogeneous candidates.
Metadata does considerable work here too. Document type, date, source system, author and page number let a query be narrowed structurally before any semantic search runs, which improves both precision and cost. The tiering and routing logic generalises from adaptive RAG, and the index sizing arithmetic from vector index sizing — noting that image embeddings often use higher dimensions than text, which inflates the index proportionally.
Where quality breaks for Indian content
Three specific weaknesses. Indian-script text within images is handled poorly by many vision-language models, because training data under-represents Devanagari, Tamil, Bengali and other scripts in visual contexts — a scanned Hindi circular is materially harder than an English one. Code-mixed audio, extremely common in Indian meetings, degrades transcription accuracy in ways that English-only benchmarks do not reveal.
And document quality: much Indian enterprise scanning is low-resolution, skewed and photographed rather than scanned, which affects both extraction and page-image embedding. The response is the same in each case — build an evaluation set from your own material, stratified by language, script and quality, and measure retrieval accuracy on it before committing to a model. Published benchmarks will overstate what you get. The document-AI parallel is in insurance claims automation.
A staged build
Start with audio, because transcription is cheap, the tooling is mature and meeting or call archives are usually the most underused knowledge an organisation holds. Add page-image retrieval for the document classes where text extraction demonstrably fails — identify those by sampling rather than assuming. Add video last, with an explicit sampling strategy and a cost estimate before ingestion begins.
For regulated Indian deployments, note that multimodal ingestion often touches more sensitive material than text pipelines do, because scanned identity documents, medical images and recorded calls are all personal data. The same residency and access-control discipline applies to the multimodal index as to the source systems, and the retrieval tier belongs in-country. The enterprise serving baseline is in right-sizing a GPU server for enterprise RAG.
Frequently asked questions
What is multimodal RAG?
Retrieval that indexes visual and audio content directly rather than only extracted text — embedding page images, captioning diagrams, transcribing recordings and sampling video frames — so that knowledge held outside clean prose becomes searchable.
Why embed page images instead of extracting text?
Because extraction discards layout, tables and diagrams. Embedding the page as an image preserves them, at the cost of more expensive embedding and a larger index. For genuinely visual document classes such as engineering and finance, that trade usually pays.
Where does the GPU budget go?
Overwhelmingly into ingestion. Processing a large document and recording archive is a batch job measured in GPU-days, while serving queries is a modest continuous load. This favours rented capacity for ingestion campaigns and owned capacity sized for serving.
How should video be handled?
Treat audio as the primary index, since transcription is cheap and spoken content carries most retrievable meaning in enterprise video. Sample visual keyframes adaptively — on scene or slide changes — rather than at a fixed high rate, which is where budgets break.
What fails on Indian content?
Indian-script text inside images, which vision-language models handle poorly due to training data gaps; code-mixed audio, which degrades transcription in ways English benchmarks hide; and low-quality photographed scans. Build a local evaluation set stratified by language, script and quality before choosing models.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.