Optimized RAG with Contextual Retrieval - Anthropic

Published on

Want to learn about an optimized way of doing RAG?

If you listen to the “IA Sob Controle” podcast, you already know that, out of all these AI companies, Anthropic is my favorite, mainly because of the papers and articles they keep publishing about their internal research.

Well, a few months ago they released yet another one of those excellent articles, this one about a technique they called “Contextual Retrieval” (which we briefly covered on IA Sob Controle 😉). It’s an optimized way to do RAG on documents that, according to them, reduced failures by up to 67%! Let’s understand the problem and how they solved it:


First, the problem: traditional RAG 🤔

RAG (Retrieval-Augmented Generation) is that technique we use to feed AIs large documents and get information from the document, and not from the “pure” AI’s knowledge. Basically, what you do to build a RAG is:

  1. Break the documents into smaller pieces of text (chunks)

  2. Turn those chunks into embeddings

  3. When someone asks a question, look for the most relevant chunks

  4. Use the question + those chunks to build a prompt for an LLM to generate the answer based on the context provided by the chunk

It works really well… until it doesn’t 😅 The problem is that when you break the document into chunks, you can lose the context of that specific chunk.

Just imagine: you have a financial report and ask: “What was ACME Corp’s revenue growth in Q2 2023?”. The system finds a snippet that says “revenue grew 3% over the previous quarter”. Cool, but… whose revenue? In which quarter? 🤷


The solution Anthropic found: Contextual Retrieval 🎯

The idea is simple, but genius: before processing the documents, they used Claude (could be another LLM of your choice) to add a little bit of context to each piece. For the example above, it would look something like:

“This snippet is from ACME Corp’s Q2 2023 financial report, where the previous quarter’s revenue was $314 million. Revenue grew 3% over the previous quarter.”

Much better, right? And the results are IMPRESSIVE:

👉 That change alone already reduced failures by 35%

👉 Combining it with BM25 (an exact-match search technique), the reduction reaches 49%

👉 Adding re-ranking, they reached a 67% improvement!


Alright, but how do you implement this in your project? 👨💻

  1. First, you need the Claude API (or another LLM of your preference) to generate the contexts. You can use a prompt like the one below:

{{WHOLE_DOCUMENT}}

Here is the chunk we want to situate within the whole document

{{CHUNK}}

Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else.


  1. Then, for each chunk of your document:
  • Generate the context using the prompt above
  • Add that context to the start of the chunk
  • Create the embeddings of this new text “augmented” with the context
  • Do the same for the BM25 index
  1. When it’s time to search:
  • Use both embeddings and BM25
  • Combine the results
  • If you can, use re-ranking on the top-k results

The cost of building this database will be a bit higher because of the context generations, but using smaller models (like Claude Haiku, Gemini Flash, GPT-mini) or open-source models, that cost ends up being pretty low. On top of that, you can use the prompt caching feature, which lets you reuse the document across API calls and will also make this cost way cheaper.


A few important tips:

💡 Experiment with different chunk sizes

💡 Test several embedding models (Gemini and Voyage did really well in the tests)

💡 Customize the contextualizer prompt for your specific case

💡 Test different numbers of returned chunks (20 seems to be the sweet spot)


So, what did you think? Has anyone implemented something similar? What were the results? Tell me here in the comments! 😃