H
Hassan Djirdeh
Guest
AI agents can plan tasks, call tools and adjust their approach as the work unfolds, but the knowledge they start with is frozen at training time. An agent that needs our current refund policy or last quarter's guidance is working from a model that has never seen either. This is the gap RAG closes: it helps AI agents retrieve current, relevant and verifiable information before they reason or act, so their decisions rest on real sources instead of model memory.
We covered the agent side of this pairing in our articles on agentic workflows and agentic RAG vs traditional RAG. In this article, we look at the relationship from the other direction: what retrieval actually does for an agent, why memory limits are necessary and what to watch for when agents start pulling data from enterprise systems.
An LLM's knowledge comes from its training data, and that data has two problems for agent work. It has a cutoff date, so anything published after training isn't there. And it's public, so internal policies, product documentation, customer records, and contracts were never in it to begin with. The model can't answer questions about information it has never seen, though it will often try, which is worse than declining.
The context window offers a partial workaround, since we can paste relevant material into the prompt, but this approach breaks down quickly. Windows are finite. Tokens cost money. And somebody still has to decide which documents matter for any particular request. For a one-off question, we can do that by hand. For an agent handling hundreds of varied requests, the selection itself becomes the hard problem.
This gap matters more for agents than for chatbots. A chatbot that improvises a wrong fact produces a bad answer. An agent that improvises a wrong fact might act on it. It could file the wrong ticket or quote the wrong policy to a customer. It could even escalate an issue based on a procedure that was rewritten last quarter. An agent grounded in model memory alone will improvise facts the moment its knowledge falls short.
Agent memory splits into two kinds, and the distinction explains where RAG fits.
Short-term memory is the working context an agent holds during a single task or session. It covers the conversation so far, intermediate results from tool calls, and the current state of the plan. Because the model itself is stateless and retains nothing between requests, all of this gets assembled and resent with every call. The context window is the hard ceiling here. A long-running task accumulates history until something has to be summarized or dropped, and whatever falls out of the window is forgotten.
Long-term memory is everything that should persist across sessions: past interactions with a user, learned preferences, resolved cases and stored facts about accounts or projects. None of this can live in the model, so it lives in external storage that our application controls. A database, a document index, a knowledge base.
Storage on its own doesn't amount to memory, though, because a stored fact does nothing for the agent until it reaches the model's context. What makes long-term memory usable is retrieval: finding the right stored information at the right moment and placing it into the context window where the model can act on it. That retrieval step is exactly what RAG provides, which is why RAG and agent memory end up being two views of the same machinery.
A basic RAG pipeline runs once. Embed the question, retrieve the closest passages, generate an answer. When an agent drives the process, retrieval becomes a repeatable step inside a larger loop.
The agent starts by recognizing an information need. Its plan calls for a fact it doesn't have, so it formulates a query, often rephrasing the user's original words into something more retrievable. It runs the retrieval, then evaluates what came back. Is the information sufficient? Is it from the right source and time period? If not, the agent rephrases, filters by metadata, or tries a different source before moving on. Only when the evidence holds up does the agent use it to decide the next action.
Consider an agent asked to prepare a renewal summary for a customer account. It retrieves the current contract terms first. Reading them, it notices a custom SLA clause, which raises a question the original request never mentioned: Has the customer filed SLA-related tickets this year? That triggers a second retrieval against the support system, and the results reshape the summary the agent was planning to write. No fixed pipeline would have made that second query, because the need for it only became visible after the first one.
That's the core difference. In traditional RAG, retrieval happens once at a predetermined point. With an agent, retrieval happens whenever the evolving plan demands it.
The accuracy gains come from grounding. When the agent's context contains the actual policy text or the actual figures, the model generates from real material rather than reconstructing facts from training patterns. Hallucination risk drops because the model has less to invent, though it never reaches zero, a point we'll return to in the FAQs.
Trust comes from traceability. Retrieved passages carry their sources with them, so an agent's answer can cite the specific document and section it relied on. When the agent proposes an action, a human reviewer can check the claim against the cited passage instead of taking the model's word for it. In our experience, this is what separates agents that teams actually deploy from agents that stall in review: nobody signs off on consequential actions backed by "the model said so."
Retrieval also keeps agents current. The agent queries the knowledge base at the moment it needs an answer, so it always works from the latest version of a document. Update a policy today, and the agent follows the new version on its next task, no retraining required.
Multi-step tasks are where agent-driven retrieval pulls ahead of the single-pass version, for a few reasons that compound.
Different stages need different information. Our renewal agent needed contract terms at one stage and ticket history at another, and neither retrieval could substitute for the other. An agent can also decompose one broad question into targeted subqueries. "How does our Q3 growth compare to guidance?" becomes one search for Q3 actuals and another for the guidance issued the quarter before, each aimed at the right document.
Retrieval across sources also lets the agent compare and reconcile. If the pricing page and the sales playbook disagree, an agent that retrieved both can flag the conflict instead of silently picking one. And because the agent evaluates results as it goes, discoveries feed back into the plan. A retrieved incident report that mentions an unresolved bug might add a whole step to the task that nobody specified upfront.
Giving an agent retrieval access to enterprise systems is a bigger decision than giving a chatbot the same access, because agents act on what they read. A few considerations come up in nearly every deployment.
None of these benefits arrive without a retrieval layer the agent can rely on, and building one from scratch means standing up extraction, chunking, embeddings, indexing and permission-aware search across every source that matters. The Progress Agentic RAG solution provides that layer as a governed service. Documents live in Knowledge Boxes, where the platform handles processing and indexing, and every answer traces back to a source passage that both the agent and its human reviewers can verify.
Agents get more than one way to look things up. The platform combines semantic retrieval, keyword search, metadata filtering and graph-based retrieval for following relationships between entities across documents, running them together as hybrid search rather than forcing a choice between them. Its retrieval agents run the agentic loop on the retrieval side, splitting questions into sub-questions and routing each to the right source, including multiple Knowledge Boxes, SQL databases and internet search. For the agents we build on top, that means retrieval that is relevant, citable and permissioned rather than a black box.
Agents need knowledge their models were never trained on, and their context windows are too small to carry everything just in case. RAG closes that gap by turning retrieval into a step the agent can repeat whenever its plan demands it, grounding each decision in current, citable sources. It's also the mechanism that makes long-term memory real, since stored facts only matter once they're retrieved back into context. The organizations getting this right pair the retrieval layer with permissions, observability, and validation before agents take consequential actions.
To give your own agents a knowledge layer they can act on, book a live demo with a Progress AI expert or start a free trial.
No. RAG reduces hallucination substantially because the model generates from retrieved text instead of reconstructing facts from memory, but the model can still misread a passage, blend two sources, or fill a gap the retrieval left open. This is why citations and validation steps matter: they make the remaining errors catchable rather than invisible.
Yes, and for real tasks it usually has to. An agent can query a document knowledge base for policy text, a SQL database for account records and web search for public information, all within one task. Platforms like Progress Agentic RAG support this directly by letting retrieval agents route sub-questions across Knowledge Boxes, databases and internet search.
Search access hands the agent a tool and hopes for the best. Agentic RAG wraps that tool in the machinery that makes it dependable: query rephrasing, hybrid search that combines semantic and keyword retrieval, metadata filtering, result evaluation and source citations, all running against governed and permissioned content. A plain search tool returns links; an agentic RAG layer returns grounded, traceable evidence the agent can act on. We compare the approaches in more depth in agentic RAG vs traditional RAG.
Continue reading...
We covered the agent side of this pairing in our articles on agentic workflows and agentic RAG vs traditional RAG. In this article, we look at the relationship from the other direction: what retrieval actually does for an agent, why memory limits are necessary and what to watch for when agents start pulling data from enterprise systems.
Why Do AI Agents Need Access to External Knowledge?
An LLM's knowledge comes from its training data, and that data has two problems for agent work. It has a cutoff date, so anything published after training isn't there. And it's public, so internal policies, product documentation, customer records, and contracts were never in it to begin with. The model can't answer questions about information it has never seen, though it will often try, which is worse than declining.
The context window offers a partial workaround, since we can paste relevant material into the prompt, but this approach breaks down quickly. Windows are finite. Tokens cost money. And somebody still has to decide which documents matter for any particular request. For a one-off question, we can do that by hand. For an agent handling hundreds of varied requests, the selection itself becomes the hard problem.
This gap matters more for agents than for chatbots. A chatbot that improvises a wrong fact produces a bad answer. An agent that improvises a wrong fact might act on it. It could file the wrong ticket or quote the wrong policy to a customer. It could even escalate an issue based on a procedure that was rewritten last quarter. An agent grounded in model memory alone will improvise facts the moment its knowledge falls short.
What Is Short-Term and Long-Term Memory for AI Agents?
Agent memory splits into two kinds, and the distinction explains where RAG fits.
Short-term memory is the working context an agent holds during a single task or session. It covers the conversation so far, intermediate results from tool calls, and the current state of the plan. Because the model itself is stateless and retains nothing between requests, all of this gets assembled and resent with every call. The context window is the hard ceiling here. A long-running task accumulates history until something has to be summarized or dropped, and whatever falls out of the window is forgotten.
Long-term memory is everything that should persist across sessions: past interactions with a user, learned preferences, resolved cases and stored facts about accounts or projects. None of this can live in the model, so it lives in external storage that our application controls. A database, a document index, a knowledge base.
Storage on its own doesn't amount to memory, though, because a stored fact does nothing for the agent until it reaches the model's context. What makes long-term memory usable is retrieval: finding the right stored information at the right moment and placing it into the context window where the model can act on it. That retrieval step is exactly what RAG provides, which is why RAG and agent memory end up being two views of the same machinery.
How Does RAG Work with an AI Agent?
A basic RAG pipeline runs once. Embed the question, retrieve the closest passages, generate an answer. When an agent drives the process, retrieval becomes a repeatable step inside a larger loop.
The agent starts by recognizing an information need. Its plan calls for a fact it doesn't have, so it formulates a query, often rephrasing the user's original words into something more retrievable. It runs the retrieval, then evaluates what came back. Is the information sufficient? Is it from the right source and time period? If not, the agent rephrases, filters by metadata, or tries a different source before moving on. Only when the evidence holds up does the agent use it to decide the next action.
Consider an agent asked to prepare a renewal summary for a customer account. It retrieves the current contract terms first. Reading them, it notices a custom SLA clause, which raises a question the original request never mentioned: Has the customer filed SLA-related tickets this year? That triggers a second retrieval against the support system, and the results reshape the summary the agent was planning to write. No fixed pipeline would have made that second query, because the need for it only became visible after the first one.
That's the core difference. In traditional RAG, retrieval happens once at a predetermined point. With an agent, retrieval happens whenever the evolving plan demands it.
How Does RAG Make AI Agents More Accurate and Trustworthy?
The accuracy gains come from grounding. When the agent's context contains the actual policy text or the actual figures, the model generates from real material rather than reconstructing facts from training patterns. Hallucination risk drops because the model has less to invent, though it never reaches zero, a point we'll return to in the FAQs.
Trust comes from traceability. Retrieved passages carry their sources with them, so an agent's answer can cite the specific document and section it relied on. When the agent proposes an action, a human reviewer can check the claim against the cited passage instead of taking the model's word for it. In our experience, this is what separates agents that teams actually deploy from agents that stall in review: nobody signs off on consequential actions backed by "the model said so."
Retrieval also keeps agents current. The agent queries the knowledge base at the moment it needs an answer, so it always works from the latest version of a document. Update a policy today, and the agent follows the new version on its next task, no retraining required.
How Does RAG Support Complex, Multi-Step Tasks?
Multi-step tasks are where agent-driven retrieval pulls ahead of the single-pass version, for a few reasons that compound.
Different stages need different information. Our renewal agent needed contract terms at one stage and ticket history at another, and neither retrieval could substitute for the other. An agent can also decompose one broad question into targeted subqueries. "How does our Q3 growth compare to guidance?" becomes one search for Q3 actuals and another for the guidance issued the quarter before, each aimed at the right document.
Retrieval across sources also lets the agent compare and reconcile. If the pricing page and the sales playbook disagree, an agent that retrieved both can flag the conflict instead of silently picking one. And because the agent evaluates results as it goes, discoveries feed back into the plan. A retrieved incident report that mentions an unresolved bug might add a whole step to the task that nobody specified upfront.
What Should Organizations Consider When Connecting Agents to Enterprise Data?
Giving an agent retrieval access to enterprise systems is a bigger decision than giving a chatbot the same access, because agents act on what they read. A few considerations come up in nearly every deployment.
- Access controls and source permissions. The agent should only retrieve what the requesting user is entitled to see. Retrieval that ignores permissions turns an agent into a data leak with a friendly interface.
- Metadata filtering and retrieval quality. Precision matters more for agents than for search pages. A human scanning results skips the irrelevant ones, while an agent may treat whatever comes back as ground truth. Filtering by document type, date or department keeps stale and off-topic content out of the loop, and retrieval quality deserves ongoing measurement rather than a one-time check.
- Governance and observability. Someone has to own which sources are authoritative, and every retrieval should be traceable: what was queried, what came back and what the agent did with it. Without that trace, debugging a bad agent decision is guesswork.
- Validation before consequential actions. Our take is that this is the non-negotiable one. Before an agent issues a refund, changes an account, or sends a customer-facing message, the evidence behind the action should be checked, by a validation step or a human, against the cited sources. Retrieval is what makes this check possible, since every claim arrives with a source passage attached, while an answer pulled from model weights leaves nothing to verify against. Skipping the check wastes the very trust that grounding was supposed to buy.
Progress Agentic RAG
None of these benefits arrive without a retrieval layer the agent can rely on, and building one from scratch means standing up extraction, chunking, embeddings, indexing and permission-aware search across every source that matters. The Progress Agentic RAG solution provides that layer as a governed service. Documents live in Knowledge Boxes, where the platform handles processing and indexing, and every answer traces back to a source passage that both the agent and its human reviewers can verify.
Agents get more than one way to look things up. The platform combines semantic retrieval, keyword search, metadata filtering and graph-based retrieval for following relationships between entities across documents, running them together as hybrid search rather than forcing a choice between them. Its retrieval agents run the agentic loop on the retrieval side, splitting questions into sub-questions and routing each to the right source, including multiple Knowledge Boxes, SQL databases and internet search. For the agents we build on top, that means retrieval that is relevant, citable and permissioned rather than a black box.
Wrap-Up
Agents need knowledge their models were never trained on, and their context windows are too small to carry everything just in case. RAG closes that gap by turning retrieval into a step the agent can repeat whenever its plan demands it, grounding each decision in current, citable sources. It's also the mechanism that makes long-term memory real, since stored facts only matter once they're retrieved back into context. The organizations getting this right pair the retrieval layer with permissions, observability, and validation before agents take consequential actions.
To give your own agents a knowledge layer they can act on, book a live demo with a Progress AI expert or start a free trial.
FAQs
Does RAG Completely Prevent AI Agents From Hallucinating?
No. RAG reduces hallucination substantially because the model generates from retrieved text instead of reconstructing facts from memory, but the model can still misread a passage, blend two sources, or fill a gap the retrieval left open. This is why citations and validation steps matter: they make the remaining errors catchable rather than invisible.
Can An AI Agent Retrieve Information from Multiple Data Sources?
Yes, and for real tasks it usually has to. An agent can query a document knowledge base for policy text, a SQL database for account records and web search for public information, all within one task. Platforms like Progress Agentic RAG support this directly by letting retrieval agents route sub-questions across Knowledge Boxes, databases and internet search.
How Is Agentic RAG Different from Giving an AI Agent Access To Search?
Search access hands the agent a tool and hopes for the best. Agentic RAG wraps that tool in the machinery that makes it dependable: query rephrasing, hybrid search that combines semantic and keyword retrieval, metadata filtering, result evaluation and source citations, all running against governed and permissioned content. A plain search tool returns links; an agentic RAG layer returns grounded, traceable evidence the agent can act on. We compare the approaches in more depth in agentic RAG vs traditional RAG.
Continue reading...