UniversalRAG: a new way of doing multimodal RAG
Published on
Check out this interesting paper that just came out for the RAG area (which I know half of you are using at work)! ;)
The paper is called “UniversalRAG: Retrieval-Augmented Generation over Multiple Corpora with Diverse Modalities and Granularities” and was published by the folks at KAIST and DeepAuto.ai.
What problem does it tackle? Current RAG systems tend to be limited: they either search only text, or, when they try to be multimodal (text, image, video), they suffer from the infamous “modality gap” (they tend to search in the same modality as the question, ignoring other sources) and don’t handle granularity well (sometimes we need a paragraph, sometimes a whole document; sometimes a snippet of a video, sometimes an entire video).
The solution of this UniversalRAG framework that they propose is the following: a “smart router” that first analyzes the question and dynamically decides:
1 - what’s the ideal modality for the answer (text? image? video? Or none, if the LLM already knows?)
2 - what’s the ideal granularity (Small chunk? Full content?)
Only after deciding that does it direct the search to the right place (the database for that specific modality and granularity). This avoids the so-called “modality gap” and fetches the information in the most suitable format.
In the benchmarks they ran (8 different ones, covering several modalities), UniversalRAG beat the simpler approaches (unimodal RAG, unified RAG).
To me, the concept makes TOTAL sense. The real world and our questions are multimodal. Having a RAG that understands not only what we ask, but where and in what format (text/image/video, short/long) to look for the answer is an important step toward smarter and more accurate AIs.
Here’s the link to the project page and the paper.
