What Is RAG Architecture? Definition and Examples

RAG, which stands for retrieval augmented generation, is a process in which LLMs use external information to expand on and improve their outputs. RAG architecture is the set of components and connections that makes RAG possible.

There are several different categories of RAG, and each one runs on its own unique architecture. In essence, the architecture of a RAG system determines the unique way in which information is stored, retrieved, and fed to an LLM.

RAG architecture is the underlying configuration of components (and their associated capabilities) that powers a given RAG system. All of these architectures build on a foundational template, often referred to as "simple" or “naive” RAG.

Here’s a high-level overview of foundational RAG architecture:

• Information source: This is where the external information is stored. It might be a search index, a vector database representing large quantities of information, or a raw document repository.
• Retrieval component: This is the mechanism by which the information source is queried. A RAG system may, for example, use a simple keyword search to find relevant chunks of text or more complex vector queries to perform a similarity search.
• Orchestration component: This layer is the part of the LLM-based application that initiates the RAG pipeline for a user prompt, either in conjunction with the LLM or independently. It also compiles the retrieved information into an expanded prompt.

RAG architecture components are usually organized into two paths. The ingestion path takes data and organizes it into a searchable information source. Vector databases, in which large amounts of text are converted into a searchable repository of vector embeddings, are one of the most widely used applications.

The retrieval path takes a user’s prompt and generates a query with which to search the information source. The orchestrator then adds the retrieved information to the context input for the LLM. For agentic and hybrid RAG architectures, there may be multiple retrieval paths.

Types of RAG architecture

It’s misleading to think of RAG in terms of completely distinct architectures. Most engineers will take a modular approach, implementing separate components to fit specific use cases. While it’s possible to draw loose boundaries, it’s important to note that there is significant overlap between architecture models.

1. Basic RAG

Basic RAG, also called “naive” or “simple” RAG, is the architectural template for other types of RAG.

The underlying process is relatively straightforward: a user enters a prompt, the orchestrator determines if external information is required (either independently or in conjunction with the LLM), and formulates and sends a retrieval query accordingly.

The external information source is commonly a vector database but may also be an SQL database, a web API, a knowledge graph, or even a “raw” document repository. This additional information is then added to the LLM’s input context.

Outside of SQL queries and APIs, there are two broad types of retrieval methods:

• Lexical search (keyword search): This is a simple form of RAG in which keywords are extracted from a user query and matched against passages of text that contain those keywords. In modern pipelines, the orchestrator may do what's called a sparse vector query, which doesn’t capture meaning, to do this.
• Dense vector search (semantic search): Dense vector embeddings capture meaning. An orchestrator can query a dense vector database in a way that allows the meaning of both the user prompt and the source information to be captured.

The use of memory is also an important part of RAG architecture. It refers to how an LLM uses context from previous chats to build a retrieval query, thereby rooting it in the broader context of a discussion (or multiple discussions). Conversational RAG is the norm for most chatbots, although some use cases, such as support ticket responses, might use single-question RAG.

2. Hybrid

Hybrid RAG combines two different types of retrieval, usually keyword and semantic retrieval. The keyword search finds matching terms, and the vector-based search finds passages with related meaning.

Other forms of retrieval, such as knowledge graph traversal or SQL database queries might also be used. The process is often guided by routing logic, either in the form of programmed rules or a model’s classification of the question (although multiple pipelines can also run simultaneously).

The orchestrator merges the results and may rerank them before passing them on to the LLM. Reranking is the process of assigning relevance scores to retrieved chunks so that the LLM prioritizes the most applicable information.

3. Adaptive RAG

In adaptive RAG, a routing component selects a retrieval strategy based on the requirements of the user prompt, and the orchestrator then executes the selected workflow.

In the original framework (Adaptive-RAG), a smaller model would act as a classifier and choose between no retrieval, single-step retrieval, or multi-step retrieval. Some architectures now use generative models or rule-based heuristics instead.

It is a popular architecture because it assigns processing power based on the complexity and requirements of a prompt, reducing unnecessary retrieval steps for simpler prompts. It’s common in assistants that deal with multiple question types.

4. Corrective RAG (CRAG)

Corrective RAG evaluates retrieved information before adding it to the LLM’s context input. A retrieval evaluator assesses whether retrieved information is relevant and useful. The unique architectural feature is the explicit evaluate-and-correct stage before answer generation.

If information is found to be incomplete or not applicable to the prompt, additional requests will be triggered. These might include different retrieval methods, such as web search or knowledge graph traversal, than were used for the original request.

Corrective RAG is common in fields where accuracy is important, such as medicine and law. It does, however, require an additional source of information that can supplement the original retrieved passages.

5. Self-RAG

In self-RAG, the model can initiate requests after receiving the initial context input. Generated answers are evaluated against retrieved evidence using reflection tokens, and further requests are sent via the orchestrator as needed. Conversely, the LLM may decide that no retrieval is required at all.

It differs from corrective RAG in that the LLM judges its own outputs, determines if retrieval is needed and if retrieved text is useful, and monitors its own answers, modifying them as needed. Corrective RAG, on the other hand, evaluates the quality of retrieved information prior to generation.

This architecture is particularly useful for complex, deep research-style prompts. Because it integrates assessments of relevance (of retrieved information) into answer generation, it is a good option for prompts that require evidence-based responses.

6. Agentic RAG

Agentic RAG refers to any form of RAG that is executed in the context of an agentic platform. The model uses its reasoning abilities to determine the type of information needed and picks from available retrieval options.

Retrieval can be triggered multiple times as required, and the model has the option to assign agents for more complex queries. For example, an LLM could request a database lookup to identify a supplier, trigger a document search to find relevant contracts, and finish with a web search to verify contact details using public sources.

Agentic RAG is the main architecture implementation in mainstream chatbots like Perplexity, ChatGPT, Claude, and so on. Because these platforms provide the core model with access to multiple tools and RAG pipelines, it can select which form of retrieval is best suited to a prompt.

7. Multimodal RAG

Multimodal RAG retrieves context across different types of media. It is an important form of RAG because it makes information available to LLMs, such as charts and audio recordings, that would otherwise be inaccessible.

The underlying architecture is the same as in any other system. For example, an indexing pipeline might prepare images for search by converting them to direct embeddings. An agentic RAG system could then retrieve these images in conjunction with semantically relevant text and feed it to a vision-language model for dual processing.

8. GraphRAG

GraphRAG uses knowledge graphs alongside vector search to retrieve information about entities. Typically, the orchestrator finds relevant starting entities, also called nodes, using keyword extraction or sometimes a preliminary vector search. A graph retriever traverses a knowledge graph to build broader context.

For example, in response to a request for a detailed breakdown of a product category, an LLM may trigger a vector search to identify specific entities (products). The orchestrator will then trigger a knowledge graph search to find related entities, such as suppliers, retail locations, reviews, and so on.

9. Branched RAG

Where a hybrid RAG architecture has the capability to use different retrieval methods, branched RAG refers to the process of running multiple queries, called branches, at the same time. Each branch can query a separate information source, or branches can query the same source in tandem with different queries or filters. Agentic RAG architecture often uses branched RAG.

10. HyDE (Hypothetical Document Embeddings)

HyDE is an innovative form of RAG in which an LLM creates a hypothetical text document of what an optimal answer might look like. This “hypothetical document” is then used to create a query for more accurate retrieval. While relevant knowledge isn’t represented in LLM parameters, the rationale is that an answer-like passage may match relevant documents better than a short question or ambiguous prompt would.

Simple RAG architecture example: Vector database

Basic RAG acts as the foundation for more complex types of architecture. In the case of a simple RAG system that uses a vector database, the source documentation is split into smaller pieces (chunks) and converted into dense embeddings. When a user enters a prompt, the orchestrator generates a query, encodes it, and queries the database to find relevant passages.

Here’s a step-by-step breakdown of the process:

• Large quantities of text into vectors using an automated ingestion pipeline and an embedding model like BERT.
• The database application organizes, indexes, and stores the vectors for fast search.
• The user enters a prompt into a chatbot (such as Perplexity).
• The application orchestrator encodes the prompt (using an embedding model) to retrieve relevant information from the vector database.
• The vector database returns relevant chunks, and the orchestrator adds these to the input context (the “large prompt”) and feeds it to the LLM.
• The LLM uses the additional context to generate an answer using its internal knowledge (stored in its parametric weights).

Advanced RAG architecture example: Agentic RAG

Agentic RAG refers to the underlying architecture of RAG systems within an agentic platform. Agentic RAG flows tend to be complex because AI agents can invoke different tools and pipelines depending on the nature of a prompt, deploying multiple agents if necessary.

Here’s a practical example of the process:

• A user enters a complex question that requires deep research into a topic. For this example, let’s say the prompt is “Provide a forecast for company earnings over the next six months in light of major geopolitical activity.”
• The LLM reasons that a mixture of RAG queries, across different content types, will be required.
• Three different agents are deployed for (a) vector search, (b) web grounding, and (c) image (multimodal) search.
• The agents search for relevant information, such as news stories and internal financial data, via the orchestrator using available retrieval tools.
• The LLM compares the retrieved results against each agent’s research question and ranks them by relevance.
• The LLM evaluates the information and triggers additional searches via individual agents where information is missing.
• Once enough information has been retrieved, the LLM generates scenario forecasts based on existing financial figures (retrieved from internal documentation) and calls a calculator tool to project earnings for each.
• The LLM generates a final answer using retrieved information and calculations, citing sources as necessary.

What’s next for RAG?

Current innovations center on bringing depth and speed to current RAG architectures, improving the accuracy of multimedia retrieval, and building systems designed to deal with up-to-the-minute information.

Here are three trends currently shaping the direction of research:

• Deep research across multiple sources: This is RAG architecture capable of planning searches, running multiple branches across disparate sources, evaluating retrieved evidence, and self-optimizing generated answers. Agentic RAG capable of executing all of these tasks is continuing to become faster and more cost-effective.
• Multimodal applications: Retrieval pipelines are getting better at working with images and video. They can increasingly pinpoint specific types of information in complex multimodal data, such as longform video and advanced diagrams.
• Grounding answers in continually changing information: LLM-based applications can draw on live information sources (such as video feeds and sensors) to provide input context. Systems can also generate multifaceted webs of information, such as with GraphRAG, to interpret this data.

RAG is foundational to how LLM-based applications work. It allows a host of different users, from everyday consumers to enterprises working in specialized fields, to overcome issues related to hallucinations, incomplete training data, and lack of access to specialized internal knowledge.

That’s why RAG systems and their underlying architecture represent one of the fastest-moving and most exciting areas of AI research.