The Agent Works. Can Quality Trust It Yet?

Six weeks without a visible error is a fact about the calendar. On its own, it doesn't tell you whether the system has earned any more trust than it had in week one.

Consider the deviation agent from Part 1. It retrieves a closed investigation from three years ago, recognises the circumstances closely resemble the case in front of it, and applies the earlier decision logic correctly. The historical record was approved and closed properly under the procedure that existed at the time. Nothing in the source is defective.

Quality rejects the answer.

The agent didn't hallucinate. Retrieval didn't fail. The record was legitimate. The problem is that the historical decision reflects a site practice Quality would no longer accept as good precedent today.

That's the Legacy Precedent Trap.

Quality systems don't hold a clean record of timeless organisational truth. They hold decisions made under earlier procedures and local interpretations, judgement calls that were reasonable in their original context and may not be reasonable now. An AI system can retrieve that history perfectly and still hand back the wrong organisational answer.

In my experience, the obvious response is to give the system more time: run it in shadow mode, let experienced SMEs review its outputs, and watch confidence build over a few months. But elapsed time doesn't create confidence. Evidence does.

One widely cited study tracked two versions of the same large language model service on identical tasks a few months apart. One model's accuracy fell by more than thirty points on a reasoning task while the other's rose by a similar margin over the same interval, then the pattern reversed on a different task. Same interval, same capability, opposite direction depending on which model and which task got measured. If researchers instrumenting the comparison directly couldn't predict which way performance would move, a Quality reviewer can't infer it from six weeks simply passing.

That's Time to Confidence, the second of the three clocks from earlier in this series. It asks what has to be demonstrated before Quality can justify relying on the system, once it's already clear the system works.

Six places a wrong answer can come from

I sort AI failures into six categories, which I use to structure that diagnosis. Applied to the deviation agent above, they look like this:

One of the six doesn't point at the AI at all.

These failures can produce almost identical outputs while needing completely different corrections. If the source is obsolete, prompt engineering doesn't repair it. If retrieval is weak, rewriting the SOP doesn't repair retrieval. If organisational context was never supplied, blaming the model's reasoning misses the actual problem. Sometimes what looks like an AI problem is really an operating-model problem that the AI happened to surface.

Which category a wrong answer belongs to should be diagnosable from how the system is built, rather than reconstructed afterwards from transcripts and guesswork. That diagnosability is an architecture decision.

It's also the part of AI governance work I spend most of my time on with clients. More on that at /ai.

What an SOP doesn't say

An SOP can describe a controlled process in detail while still missing everything an experienced practitioner uses to decide a hard case. In deviation management, the procedure defines roles, classifications and timelines, while a seasoned Quality lead also judges whether evidence is enough to accept a root cause, which historical investigations still count as credible precedent, and how much residual risk the organisation will tolerate. Some of that logic lives only in the judgement of people who've run the process for years.

I call that layer Quality Intent: the organisation's intended interpretation of how a regulated process should produce a defensible Quality decision, including its evidence expectations, decision boundaries, exceptions, risk tolerance and escalation points. It isn't a regulatory term, and it doesn't replace procedures, validation or Quality Risk Management. It's a working concept for locating the decision logic an AI system needs, the kind the organisation may never have written down.

That reframes the question worth asking after an SME disagreement. Rather than "how do we make the system give the answer the SME expected," the better question is what defensible decision rule would make either answer correct. If that rule can be named, the fix belongs in source, retrieval, context, instruction or reasoning. If it can't, testing has found something more useful: a decision boundary the organisation hasn't resolved clearly enough to hand to anyone, human or AI.

Manufacture disagreement on purpose

Thirty days in shadow mode can produce useful evidence, but thirty days alone prove nothing. A system might spend that month on thirty variations of the same easy case, while a second gets challenged in five days with conditions built to expose where it stops being dependable. The second exercise tells Quality far more.

Consider a qualification set built around the situations that actually break Quality decisions: conflicting procedures never formally reconciled, incomplete records, historical decisions no longer regarded as good precedent, and cases where experienced reviewers disagree. The deviation agent's failure is exactly that shape: a document trail saying one thing and a lived process that had quietly moved on to another.

A system shown only clean, single-answer cases produces clean, confident answers. That result says something about the test set. It says very little about the record the system will eventually have to handle once conditions stop being clean.

The correction is evidence too

Suppose a system generates 500 outputs during controlled testing and Quality materially corrects 42 of them. An acceptance rate tells you something; those 42 corrections usually tell you more. Each should leave a record answering five questions: what was wrong, which of the six categories caused it, what needs to change, whether the change affects intended use or validated behaviour, and what evidence shows the underlying failure was actually addressed, beyond the one example in front of you.

That last distinction matters. Correcting the example in front of you proves the example was corrected. Confidence requires evidence the failure itself has been addressed, without introducing unacceptable behaviour somewhere else.

I keep this as a standing template: five fields, filled in every time an SME makes a material correction, however minor it looks at the time.

Not every wrong answer is model behaviour

Most enterprise AI systems don't produce an answer from a model in isolation. The output reflects a chain: source information, retrieval, supplied context, instructions, the model itself, and workflow rules, all before a regulated user sees the result. A defect anywhere in that chain can surface as a plausible but unacceptable answer.

The clearest evidence for the source-and-retrieval half of that chain comes from outside GxP, from health question-answering research rather than a deviation and a CAPA; the parallel to regulated Quality work is my own reading, and the researchers make no claim about GxP. A single adversarial document pushed an otherwise accurate system's answers down near zero, and adding a helpful document to the same pool only partially recovered it, to roughly 40 to 45 percent for the worst content tested, well short of what a helpful document reaches alone. The reason traces back to ranking rather than reasoning: when documents were ranked rather than hand-picked, adversarial content dominated 92 to 94 percent of the top ten results returned for a query, and the ranking model simply surfaced whatever scored highest.

What the validation package actually needs to prove

This sits comfortably with how GAMP already asks for evidence proportionate to a system's risk and intended use. GAMP 5 Second Edition covers AI and machine learning in Appendix D11; the standalone ISPE GAMP Guide: Artificial Intelligence, published in July 2025, extends those principles to AI-enabled GxP systems. The failure map and Quality Intent aren't requirements derived from that guidance. They're the practitioner tools I use to answer a question GAMP doesn't answer for you: when an AI-supported Quality decision is wrong, what exactly failed, and what evidence would justify relying on the system again?

For a validation summary to be useful in defending reliance on the system, I'd expect it to make several things explicit: the intended use the system was built for, its performance against a challenge set built to be hard, its known limitations, the human oversight design that catches what it gets wrong, the escalation boundaries that define when a person takes over, the failure conditions the organisation accepts, and the evidence behind releasing it on those terms. General good performance over a stated number of weeks doesn't appear anywhere on that list.

There's no universal number of test cases that creates confidence either, because the evidence threshold follows the intended level of reliance. An agent surfacing records for a trained reviewer carries different risk from one recommending a deviation classification outright; a system whose draft always gets substantive human review sits differently from one whose output becomes the decision itself. The question worth asking is less "is the AI validated" and more specific: what is the organisation prepared to rely on this system to do, and what evidence supports that reliance?

Confidence is evidence-triggered, not calendar-triggered. The failure map, the challenge set, the correction record: all three exist to give Quality something to trigger on besides the date.

If a capability can be established in four weeks, as the first piece in this series described, Quality doesn't need six months of passive watching. It can bring the hard cases forward immediately: contradictory evidence, obsolete precedent, incomplete records, the cases its strongest SMEs still debate. Rapid capability is the chance to start generating that evidence sooner.

Once the failure patterns are understood and Quality has enough evidence to justify a defined level of reliance, a different clock starts to matter. Someone will correct the agent, and the question becomes where that correction goes, and what it takes to improve the system without losing control of the evidence supporting reliance on it. That's Time to Maturity, and it's where Part 3 of this series picks up.

The full series

More from Navata Insights → /insights