Ask a question about Redis and the job seems simple: find the right documentation and explain it. But documentation questions come in many shapes.

“What does XACK do?” is a precise command question. “How should I think about eviction policies?” is a broader concept question. “Why is my stream consumer falling behind?” may span several pages, commands, and ideas.

We set out to build an agent interface that could answer Redis questions with precise, cited evidence from redis.io/docs. This post follows how we evolved from one broad document entity to multiple entities, linked by explicit relationships that help the agent move from discovery to evidence. Our evaluations revealed stronger retrieval overall, but at the cost of more tool calls and slower tail latency.

Starting simple

Context Retriever lets developers model their data like an ORM. An “entity” defines a schema for data; Context Retriever exposes retrieval tools that an agent can call. The application still decides which tools to give the agent and how retrieved evidence should be used in the final answer.

For V1, we started with one entity to represent a documentation page, section, or chunk of text: RedisioDoc. It carried a stable ID, title, topic, section, content, vector embedding, and canonical URL.

Below is the data model for records derived from documentation:

The above schema generated six tools, but our application only exposed four of them to the agent:

This was a useful baseline. A small toolset made the system easier to inspect while we improved documentation coverage, stable IDs, citations, query handling, and result size.

But the simple model also flattened several retrieval jobs into one shape. Whether a question needed a page overview, an exact command reference, or a precise passage, the agent searched the same entity and had to sift through the results itself. Our evaluations eventually showed where that approach began to break down.

When a good answer hid weak retrieval

At first, we judged the system as a user would.

Did the answer sound right? Did it cite relevant pages? How long did it take?

Those questions matter, but they miss a problem. A strong foundation model can turn incomplete evidence into convincing prose, using general knowledge to fill gaps, so a fluent answer can sit on weak retrieval.

That matters even when the final answer looks fine. If retrieval is weak, the model has to read more irrelevant text, make more calls, and rely more on what it already knows. That costs latency and tokens, and it pushes you toward a larger model to compensate. Good retrieval lets a smaller, faster model produce the same answer from evidence it can cite.

We therefore began evaluating the evidence, with hard metrics, before the model wrote the answer. This gave us a separate view of retrieval quality: What pages did the agent find? Where did they rank? Which tools led it there?

Evals over golden data

To understand whether retrieval was improving, we needed to measure the evidence before the model turned it into an answer. We built a standalone evaluation kit that could test different retrieval agents, MCP servers, APIs, or saved traces against the same documentation corpus.

[Image placeholder] Evaluation kit workflow diagram (01 Set up a new retrieval target → 02 Run the shared evaluation). Upload from the source draft. Suggested alt text: “Two-phase evaluation workflow: an agent skill writes a target-specific adapter, then a reusable harness scores the adapter’s RetrievalTrace against a golden dataset.”

The visual above shows the workflow in two phases.

First, we connect a retrieval target to the kit. An agent skill guides this setup by inspecting representative raw outputs and creating a small, target-specific adapter. The adapter knows how to call that system, parse its response, preserve the original result order, and map its source identifiers into stable documentation page IDs. It contains the system-specific integration logic, but no scoring logic.

Once the adapter is validated, the shared evaluation can run. For each row, the golden dataset supplies:

A question.
The primary documentation sources expected to answer it.

The questions were divided into four groups that represented different retrieval patterns:

Simple lookups asked for one clear concept or command. For example, “What does BITCOUNT do?” expected the BITCOUNT command page as its primary source.
Multi-step questions required evidence from several sources. “How do I create a Bloom filter, add items, and check membership?” expected the BF.RESERVE, BF.ADD, and BF.EXISTS command pages.
Hard cases included troubleshooting questions, ambiguous terminology, and questions that crossed several parts of the documentation. For “XACK did not reduce stream memory. What should I trim or delete?”, the expected sources included the XACK, XTRIM, and XDEL command pages and the Streams guide.
Chat-style questions reflected shorter, production-like prompts. “How do I monitor Redis latency?” expected the Redis latency guide.

The reusable harness sends the question through the adapter, which invokes the retrieval target. The adapter converts the raw response into a common RetrievalTrace: an ordered list of retrieved page IDs, together with the tool calls, errors, and latency recorded during retrieval.

The harness then removes duplicate pages without changing their original order and compares the ranked results with the expected sources. This lets us measure both whether the right evidence appeared and how early it appeared. Each run produces a detailed trace.jsonl for inspecting individual questions and a report.md containing metrics such as nDCG, recall, and hit rate.

Separating the adapter from the harness made the kit reusable. To evaluate another agent or MCP over the same corpus, we write a new adapter while keeping the golden dataset and scoring process unchanged. It also keeps two questions separate: did the agent retrieve the right evidence, and did the model use that evidence to produce a good answer?

The problem with one broad “document”

The trace pointed to a problem with the shape of our context model.

One trace made the problem clear. When asked, “What does BF.RESERVE do?”, V1 searched the full documentation collection four times. Its first results included Kubernetes and REST API pages that happened to contain words such as “reserve” or “reservation.” The Bloom filter guide appeared only near the bottom of the first ten results, while the BF.RESERVE command page never appeared. The problem was not a lack of documentation; the retriever did not have a direct path to the command as an entity.

V1 represented pages, sections, and command references through one broad entity: RedisioDoc. The agent could search it by text or meaning, filter by a broad topic, or fetch a document if it already knew the ID. That worked for many questions, but it treated different retrieval tasks as if they were the same. A question might require:

An exact command reference.
A page overview to find the right topic.
A focused passage to support the answer.
More evidence from the page where a useful passage was found.

These tasks need different kinds of information. A short passage is useful evidence, but may lack context. A page summary helps with discovery, but may be too broad to cite. A command has a clear name and structure, so searching for it as general prose can introduce unrelated matches.

That is what happened with BF.RESERVE. The agent knew the name of the command, but V1 could not treat it as a command and go directly to its record. It searched the wider documentation collection instead, where “reserve” also appeared in Kubernetes and REST API pages.

Three entities connected by retrieval paths

The BF.RESERVE trace suggested a more specific solution, where we stop asking one entity to handle every kind of retrieval. In our next design, we split the documentation into three entities:

RedisioChunk holds focused passages used as answer evidence.
RedisioPage holds page summaries used for discovery.
RedisioCommand holds structured command references for precise lookup.

This gave the agent better choices, but the entities still needed to work together. We also declared an explicit relationship link between pages and chunks, and between pages and commands. These relationships used the stable page_id shared by the records.

[Image placeholder] Entity relationship diagram (RedisioPage ↔ RedisioChunk ↔ RedisioCommand, both relationships join on page_id). Upload from the source draft. Suggested alt text: “RedisioPage for broad discovery links to RedisioChunk for answer evidence and RedisioCommand for exact command lookup, joined on page_id.”

The added relationships made joins part of the retrieval model and the agent could now follow clearer paths. For example:

For a direct question, search chunks and return the strongest passages.
For a broad question, discover a page from its summary, take its page_id, and retrieve chunks from that page.
For a command question, fetch the command record directly, then retrieve related prose when more explanation is needed.

The agent could now choose a route based on the question instead of treating every request as the same kind of document search. The shift was not simply from one index to three indexes. We gave the agent a small map of the documentation: distinct places to search and stable paths for moving between them.

What the results showed

We compared V1 and V2 on the same 62-question retrieval set. We used gpt-5.4-mini to power the retrieval agent, and both had the same recursion limit of 16 (i.e. the agent can only take a maximum of 16 steps for one question before it hits the safety cap and is stopped).

Our main ranking metric was nDCG@5. Recall only asks whether an expected page appeared somewhere in the results. nDCG also rewards placing it near the top, and it gives more credit to primary sources than to related ones, which matches how our golden dataset is labeled. Rank matters here because the agent reads results in order, and a correct page buried at position nine is much less likely to be used or cited than one at position one. We report recall@5 alongside it to show whether the right evidence appeared at all.

In simple terms, nDCG measures whether the expected documentation appeared near the top of the first five results. Higher is better. V2 was stronger overall:

The latest V2 run improved retrieval across every question group. The largest gain came from simple lookups, where exact command and entity-specific retrieval paths helped the expected sources reach the top results. Multi-step questions also improved strongly, suggesting that the agent was better able to gather evidence from more than one part of the documentation.

The improvement also held beyond the first five results. Overall nDCG@10 increased from 0.493 to 0.658, while recall@5 increased from 0.610 to 0.743.

The richer retrieval surface required more work from the agent:

V2 found and ranked better evidence, but it made more tool calls and took longer. That extra latency matters most in the slowest requests. The public ask path has a 25-second request limit. At p95, V1 used about 40% of that budget, while V2 used about 72%. That left less room for network delays, provider variance, or another retrieval step. Therefore, the next step is not to undo the richer model, but to make navigation more efficient; for example, by reducing unnecessary tool calls and ending retrieval once enough evidence has been found.

What we learned so far

Relationships turn retrieval into navigation

Splitting the corpus into pages, chunks, and commands gave the agent better places to search. Connecting them made each result useful as a next step.

A page search did not need to end with a page summary. Its page_id could be passed to the chunk filter to retrieve evidence from that page. A command record could use the same page identity to connect precise command details with supporting prose.

Without those paths, the agent has to infer the connection, construct the right identifier, and remember the join from its prompt. Explicit relationships move that knowledge into the retrieval model. The solution design does not choose or execute the route for the agent, but it makes the available route clear and dependable.

This was an important shift in how we thought about retrieval quality. A good result is not only relevant; it should also make the next useful retrieval move easier.

Model retrieval around real user intent

Chunking alone does not create a retrieval strategy. A command question, a broad discovery question, and a request for supporting evidence need different result shapes. Modeling those shapes as separate entities gave the agent a way to choose what it needed before searching, instead of sorting out the meaning of broad results afterward.

Stable IDs are core infrastructure

Relationships only work when both sides agree on identity. Stable page, section, chunk, command, anchor, and URL IDs allowed results from one tool to become inputs to another. They also made citations, evaluation, and debugging more reliable. We would design this identity system earlier next time.

Retrieval and answer quality need separate tests

Users experience the final answer, so answer quality still matters. But evidence-level tests tell us whether a change to the schema, index, or routing actually improved retrieval. Using both views keeps a fluent model from hiding weak search.

Better retrieval can require more retrieval

V2 improved ranking overall, but it also increased tool use and slow-tail latency. The next step is to use its tools more selectively: route command questions directly, use page discovery only when needed, and set deterministic criteria for the LLM to stop once the evidence is strong enough.

A better map of the docs

We began by asking how to retrieve the right Redis document.

We ended up with a better question: what map of the documentation helps an agent take the right next step?

Sometimes that step is finding a passage. Sometimes it is discovering a page. Sometimes it is looking up a command. Sometimes it is following a link from a broad result to evidence ready for citation.

Context Retriever was the mechanism we used to model the entities and relationships, but the broader lesson is not to add more entities or tools by default. It is to model the few retrieval moves that match the real structure of the corpus, connect them with stable identifiers, and measure whether the agent actually uses them well.

The move from one document type to a connected documentation surface did not remove complexity. It made that complexity easier to see, measure, and improve.

For agentic retrieval, that may be the most useful result of all.