A
Adam Bertram
Guest
Why does your RAG app answer a two-part question with a confident paragraph that answers neither part? Say a user asks which vendor missed its SLA most recently, and whether that vendor was also flagged for a compliance issue that quarter. What comes back reads well and settles nothing, and the model isn’t the reason. The question needs two facts, and they sit in different documents that never mention each other, so one search pass can’t travel from the first to the second. Reaching that second fact takes a second search, one built from what the first search found. That’s multi-hop retrieval: a search strategy that runs several passes in sequence, each query shaped by what the last one returned.
A single vector search embeds the whole question as one vector and ranks passages by how close they sit to it. A question that connects two facts produces an embedding sitting near passages about each subject and near nothing that carries the connection between them. The document holding the second fact usually shares almost no vocabulary with the question as asked, so you can’t search for it until you already know the first answer, which is exactly what one pass doesn’t have.
MultiHop-RAG was built to measure how badly single-pass retrieval handles those questions: 2,556 queries over a news corpus, with the evidence for most queries spread across two to four separate documents. Standard embedding models and commercial LLMs both scored poorly on it, which is the useful part: this isn’t a tuning problem.
The failures cluster into shapes worth recognizing in your own query logs:
None of those three shapes is a prompt problem: if the evidence never reaches the model, no instruction rewrite improves the answer.
Instead of embedding the question once, a multi-hop system splits it into sub-questions, retrieves for the first one, and folds what came back into the query it writes for the second. DSPy’s multi-hop tutorial makes the state that travels between hops explicit by pairing two modules: one generates the next search query, the other appends notes drawn from the passages just retrieved. Hop two can then search for a term the original question never contained, because hop one supplied it.
Self-ask prompting formalizes the same move at the model layer. Before answering the question it was handed, the model asks itself a follow-up, answers that, then uses the answer to finish the original.
Every hop expands the search space, so a chain needs a stop condition or it’ll happily keep going. That means a hop budget, plus a check after each pass on whether the evidence now answers the question or has stopped improving. Hop count is a knob you own, not a number the system discovers on its own.
Chain the hops the obvious way and the error compounds. Each pass reformulates the accumulated context into a fresh query and searches again, squeezing the running state back into a single vector every time. Whatever falls out of that vector isn’t coming back. Pull one distractor at hop one, a passage that sits close in embedding space but beside the point in fact, and every query after it inherits the detour.
Baleen attacks the compression side. After each hop it condenses the retrieved passages into one compact summary and carries that summary forward, so the next query is conditioned on stated facts rather than an ever-longer, ever-blurrier input. On HotpotQA, which requires reasoning over exactly two supporting documents, it reached 96.3% answer recall at 20 retrieved passages.
Beam Retrieval attacks the commitment side. Rather than betting the chain on one passage per hop, it keeps several candidate chains alive and trains the retriever across all hops jointly, so an early wrong guess kills one branch instead of the whole retrieval. Condensation keeps the latency down. Beams buy you recall on the queries where hop one is genuinely ambiguous.
The obvious objection first: context windows are enormous now, so why not skip the chaining and hand the model everything a broad search returns? Because position still matters. Research on long-context models found accuracy drops measurably when the fact an answer depends on sits in the middle of the context rather than near either end. A bigger window doesn’t fix that, so order the evidence before it reaches the model.
The stop condition deserves better than a hard-coded hop count too. Self-RAG trains the model to grade its own retrieved passages and decide whether another retrieval is warranted, which turns “do we have enough yet?” into a learned signal instead of a number you picked in a config file and haven’t revisited since.
None of those mechanisms survives contact with a corpus that keeps changing, and that’s where maintenance actually lives. Documents get added, chunking strategies get retuned, and a chain that resolved cleanly in March starts answering past its evidence by June. That decay is the part a chain can’t self-report, so production platforms, including Progress Agentic RAG, build evaluation into the pipeline itself.
A basic RAG pipeline inside Progress Agentic RAG makes one pass: the question goes to semantic search, the results get augmented, an LLM writes the answer. Retrieval Agents replace that single pass with a configurable workflow, and it’s built from two pieces. Drivers connect the sources a question can be answered from, including indexed document corpora, SQL databases, internet search services and MCP servers. The workflow decides how a question moves between arrival and answer.
The hops live in the workflow. A question that connects two facts gets decomposed into sub-questions, each routed to the driver that can actually answer it. Routing per sub-question matters more than it sounds: hop one can run against your indexed documents while hop two runs against a SQL driver, so two facts don’t have to be co-located in one store to get chained together. A homegrown chain usually hops within one index, so the moment the second fact lives somewhere else, it’s done.
Evaluation is what closes the loop. REMi, the platform’s own RAG evaluation model, runs on a regular basis and charts the three scores it produces over time. Two of its scores map onto the ways a chain fails. Groundedness catches an answer that ran past the evidence the chain retrieved. Context Relevance catches a hop that retrieved confidently and wrongly. Start with those two scores: instrument the query shapes already sitting in your logs, watch which one moves, and add hops where the evidence says you need them rather than everywhere.
Published multi-hop benchmarks mostly top out between two and four hops, a reasonable ceiling to start from. Every hop adds a retrieval round trip and usually an LLM call, so latency and cost climb while the evidence each hop contributes falls off. Set a hard cap, then let a sufficiency check end the chain early on questions that resolve in one pass.
No. A graph gives you explicit relationships to traverse, which narrows the search space when your domain has stable entity relations. Chaining works over plain text without one, because the connection between hops is carried by what hop one retrieved rather than an edge somebody modeled in advance. Build the graph when the relations are worth maintaining on their own merits.
Take the questions your current pipeline answers badly and check where their supporting evidence lives. When it consistently sits across several documents, prompt work and re-ranking won’t close the gap. When it sits in one document the pipeline keeps missing, that’s usually a chunking or ranking problem, and another hop won’t help.
Continue reading...
Where Does Vector Search Fall Short for AI Answer Generation?
A single vector search embeds the whole question as one vector and ranks passages by how close they sit to it. A question that connects two facts produces an embedding sitting near passages about each subject and near nothing that carries the connection between them. The document holding the second fact usually shares almost no vocabulary with the question as asked, so you can’t search for it until you already know the first answer, which is exactly what one pass doesn’t have.
MultiHop-RAG was built to measure how badly single-pass retrieval handles those questions: 2,556 queries over a news corpus, with the evidence for most queries spread across two to four separate documents. Standard embedding models and commercial LLMs both scored poorly on it, which is the useful part: this isn’t a tuning problem.
The failures cluster into shapes worth recognizing in your own query logs:
- Comparison. Two entities and one judgment: which vendor missed more often, which region lags.
- Ordering. Events whose sequence only appears after you pull each event’s date from a different record.
- Inference. A fact nobody wrote down, which exists only as the product of two facts somebody did.
None of those three shapes is a prompt problem: if the evidence never reaches the model, no instruction rewrite improves the answer.
How Does Multi-Hop Retrieval Work?
Instead of embedding the question once, a multi-hop system splits it into sub-questions, retrieves for the first one, and folds what came back into the query it writes for the second. DSPy’s multi-hop tutorial makes the state that travels between hops explicit by pairing two modules: one generates the next search query, the other appends notes drawn from the passages just retrieved. Hop two can then search for a term the original question never contained, because hop one supplied it.
Self-ask prompting formalizes the same move at the model layer. Before answering the question it was handed, the model asks itself a follow-up, answers that, then uses the answer to finish the original.
Every hop expands the search space, so a chain needs a stop condition or it’ll happily keep going. That means a hop budget, plus a check after each pass on whether the evidence now answers the question or has stopped improving. Hop count is a knob you own, not a number the system discovers on its own.
Why Naive Chaining Breaks and What Fixes It
Chain the hops the obvious way and the error compounds. Each pass reformulates the accumulated context into a fresh query and searches again, squeezing the running state back into a single vector every time. Whatever falls out of that vector isn’t coming back. Pull one distractor at hop one, a passage that sits close in embedding space but beside the point in fact, and every query after it inherits the detour.
Baleen attacks the compression side. After each hop it condenses the retrieved passages into one compact summary and carries that summary forward, so the next query is conditioned on stated facts rather than an ever-longer, ever-blurrier input. On HotpotQA, which requires reasoning over exactly two supporting documents, it reached 96.3% answer recall at 20 retrieved passages.
Beam Retrieval attacks the commitment side. Rather than betting the chain on one passage per hop, it keeps several candidate chains alive and trains the retriever across all hops jointly, so an early wrong guess kills one branch instead of the whole retrieval. Condensation keeps the latency down. Beams buy you recall on the queries where hop one is genuinely ambiguous.
How to Get the Chain Into Production Without Babysitting It
The obvious objection first: context windows are enormous now, so why not skip the chaining and hand the model everything a broad search returns? Because position still matters. Research on long-context models found accuracy drops measurably when the fact an answer depends on sits in the middle of the context rather than near either end. A bigger window doesn’t fix that, so order the evidence before it reaches the model.
The stop condition deserves better than a hard-coded hop count too. Self-RAG trains the model to grade its own retrieved passages and decide whether another retrieval is warranted, which turns “do we have enough yet?” into a learned signal instead of a number you picked in a config file and haven’t revisited since.
None of those mechanisms survives contact with a corpus that keeps changing, and that’s where maintenance actually lives. Documents get added, chunking strategies get retuned, and a chain that resolved cleanly in March starts answering past its evidence by June. That decay is the part a chain can’t self-report, so production platforms, including Progress Agentic RAG, build evaluation into the pipeline itself.
How Does Progress Agentic RAG Incorporate Multi-Hop Retrieval Into Its Architecture?
A basic RAG pipeline inside Progress Agentic RAG makes one pass: the question goes to semantic search, the results get augmented, an LLM writes the answer. Retrieval Agents replace that single pass with a configurable workflow, and it’s built from two pieces. Drivers connect the sources a question can be answered from, including indexed document corpora, SQL databases, internet search services and MCP servers. The workflow decides how a question moves between arrival and answer.
The hops live in the workflow. A question that connects two facts gets decomposed into sub-questions, each routed to the driver that can actually answer it. Routing per sub-question matters more than it sounds: hop one can run against your indexed documents while hop two runs against a SQL driver, so two facts don’t have to be co-located in one store to get chained together. A homegrown chain usually hops within one index, so the moment the second fact lives somewhere else, it’s done.
Evaluation is what closes the loop. REMi, the platform’s own RAG evaluation model, runs on a regular basis and charts the three scores it produces over time. Two of its scores map onto the ways a chain fails. Groundedness catches an answer that ran past the evidence the chain retrieved. Context Relevance catches a hop that retrieved confidently and wrongly. Start with those two scores: instrument the query shapes already sitting in your logs, watch which one moves, and add hops where the evidence says you need them rather than everywhere.
FAQ
How Many Hops Is Too Many?
Published multi-hop benchmarks mostly top out between two and four hops, a reasonable ceiling to start from. Every hop adds a retrieval round trip and usually an LLM call, so latency and cost climb while the evidence each hop contributes falls off. Set a hard cap, then let a sufficiency check end the chain early on questions that resolve in one pass.
Do I Need a Knowledge Graph to Run Multi-Hop Retrieval?
No. A graph gives you explicit relationships to traverse, which narrows the search space when your domain has stable entity relations. Chaining works over plain text without one, because the connection between hops is carried by what hop one retrieved rather than an edge somebody modeled in advance. Build the graph when the relations are worth maintaining on their own merits.
How Do I Tell Whether My Pipeline Actually Needs Multi-Hop Retrieval?
Take the questions your current pipeline answers badly and check where their supporting evidence lives. When it consistently sits across several documents, prompt work and re-ranking won’t close the gap. When it sits in one document the pipeline keeps missing, that’s usually a chunking or ranking problem, and another hop won’t help.
Continue reading...