A
Adam Bertram
Guest
Article Summary
Continuous RAG evaluation turns accuracy from a sampled opinion into scored, traceable evidence for governance and quality teams. Instead of one reviewer’s read on a handful of answers, every response gets scored for context relevance, answer relevance and groundedness, and those scores accumulate into a trend line a quality lead can actually inspect. Accuracy stops being a launch-day impression and becomes a standing control you can point to.
A RAG answer that looked fine last week is not a control.
The production question is sharper: can the team prove why an AI answer was trusted, which source supported it and what changed when accuracy drifted? Manual spot-checking rarely leaves that kind of evidence. Continuous evaluation does.
Why Spot-Checking Is Not a Defensible Control
A manual review usually starts with good intent. Someone reads a few answers, decides they sound relevant and signs off. For a pilot, that can feel practical. For production AI, the method leaves too much unmeasured.
Scale and consistency are the immediate problems. A retrieval-augmented generation (RAG) system can drift when source content changes, retrieval ranking shifts or users ask questions outside the original test set. That gap widens at scale: many enterprise AI programs run through one central data or AI team, and no central team can hold a domain-specific quality bar for every part of the business the system serves. When a finance answer and a legal answer do not share the same definition of “good,” hand-checking gives you no consistent way to judge either. One reviewer may value a concise answer, while another expects citations, caveats or a fuller explanation. A handful of sampled answers will not show where drift started or give governance a repeatable measurement.
The deeper problem is evidence. If a reviewer asks what was checked and whether yesterday’s answer still holds today, a spot-check has an awkward answer: someone looked. That is thin paper for an audit file.
Accuracy needs to become a measured signal on every answer, not a sampling exercise at the end of a project.
The Three Scores That Make Accuracy Measurable
The useful question is not simply, “Was the answer good?” That question hides the cause — and at scale, neither hand-checking nor an automated judge answers it on its own. A RAG answer can fail because the system retrieved the wrong material, because the model ignored good material or because the answer sounds right while leaning on facts that were never retrieved.
REMi, the evaluation layer in Progress Agentic RAG that scores every response for relevance, accuracy and groundedness, separates those cases into distinct signals.
- Context Relevance shows whether retrieval brought back context that fit the user’s query.
- Answer Relevance focuses on the response itself: did it answer the question asked?
- With Groundedness, the test is source support — whether the answer can be traced back to the retrieved context.
Those metrics matter because each answers a different governance question. Groundedness is the hallucination check. If an answer says a policy allows something, the evaluation question is not whether the sentence sounds plausible. It is whether the retrieved source material supports that claim.
The review conversation changes. Instead of asking a subject matter expert to reread random answers, a quality lead can inspect the score that moved, the source that was retrieved and the answer that lost support.
How Score Patterns Point to the Failure
The three scores are most useful when read together. A single low score says something went wrong. The pattern tells you where to look.
| Score Pattern | When It Usually Means | Governance Question |
| High Answer Relevance, low Context Relevance and low Groundedness | The answer addressed the question, but the system did not retrieve support for it. | Did the model fill the gap from memory? |
| High Context Relevance, low Answer Relevance and low Groundedness | The right material was retrieved, but the answer did not use it well. | Is the generation step dodging or summarizing poorly? |
| High Groundedness, low Context Relevance and low Answer Relevance | The answer stayed close to retrieved text, but retrieved the wrong text. | Did the system confidently answer the wrong question? |
That is the difference between quality theater and diagnosis. If retrieval missed, you investigate chunking, metadata, ranking or how the query was rewritten. If generation failed, you look at the prompt, model behavior or response policy. If the system grounded itself in the wrong source, you have a retrieval problem wearing a compliance-friendly outfit. The answer looks traceable. The trace still points to the wrong place.
For an AI quality engineer, the value is speed. For a governance lead, the value is defensibility. A finding that says “bad answer” is a complaint. A finding that says “high answer relevance, low groundedness” is a record of the failure mode.
Continuous Monitoring Makes It a Standing Control
One clean evaluation pass at launch does not prove the system will stay accurate. A knowledge base changes. Source documents get replaced. Users stop asking demo questions and start asking the messy ones that matter. The system that looked healthy at rollout can drift without anyone touching the model.
That is why continuous scoring matters. Progress documents REMi performance views that track quality over rolling seven-day and 30-day windows. Those windows turn isolated judgments into a trend line. If Context Relevance drops after a content sync, the quality review can start with retrieval evidence instead of a generic debate about whether AI is “getting worse.”
The same loop can expose knowledge gaps. Unanswered-question tracking shows where users asked for information the system could not retrieve or support. That gives the content owner a remediation backlog: add missing documentation, improve indexing, review the next score window and verify whether the gap closed without rebuilding the whole pipeline.
The audit record should connect the source update, the sync event, the retrieved context, the generated answer, the REMi scores and the reviewer decision. That chain lets a governance lead explain the input path before defending the AI outcome.
Automated evaluators are not magic, and they should not be treated as final truth. A scoring model can be miscalibrated. A metric can miss a domain-specific nuance. The answer is not to go back to gut feel; the answer is a visible control loop. Define the score threshold, name the owner, open an exception or remediation record when the threshold is crossed, and require review before expanded use continues.
Manual review still has a job. It should investigate the cases the scores surface and validate whether the measurement aligns with business risk. It should not be the only thing standing between a production AI system and users who assume the answer is safe to trust.
Accuracy Has to Leave Evidence Behind
A production RAG system earns trust by proving its work repeatedly. Launch approval is only the first checkpoint; standing evaluation is the control.
Continuous evaluation gives governance and quality teams a shared language for that proof: retrieval fit, response fit and source support. The labels are different because the failures are different.
Those are not abstract quality labels. They are evidence fields for deciding whether an AI experience deserves broader usage and more responsibility.
If you are evaluating RAG for production use, ask for the scores behind the answer, the trend after source changes and the control that runs when a score drops. To see the measurement loop, start with the REMi evaluation model or book a demo.
FAQ
Does Continuous Evaluation Replace Human Review?
No. It changes where human review spends its time. Instead of reading random answers and hoping the sample represents production, reviewers can investigate score patterns, calibrate thresholds and focus on cases where quality risk is visible.
Which REMi Metric Matters Most for Governance?
Groundedness is usually the most audit-relevant metric because it asks whether the answer is supported by retrieved context. Context Relevance and Answer Relevance still matter because they explain whether the failure came from retrieval, generation or a mismatch between the two.
What Should a Team Do When Scores Drop?
Start with the metric that moved. A Context Relevance drop sends the team toward retrieval, indexing, chunking or metadata. When Groundedness falls, look for unsupported answers; when Answer Relevance falls, review the generation step and how the question is being interpreted.
Continue reading...