How to OQ/PQ a Hallucination

What qualification evidence looks like when a RAG assistant can produce a different answer every time.

In eighteen years of qualifying systems in regulated environments, I have watched validation follow the same shape every time. I define the requirements. I write test scripts with expected results. I execute them, the actual results match the expected results, and I sign. Operational Qualification confirms the system does what the specification says under defined conditions. Performance Qualification confirms it keeps doing it in the live process. The whole discipline rests on one quiet assumption: that the same input, run again, produces the same output.

A retrieval-augmented generation assistant sets that assumption aside by design. Ask it the same question twice and you may get two differently worded answers, both correct, or two differently worded answers, one of which is not. Variation is no longer evidence of failure. So the test script that reads “expected result: the following paragraph” has nothing to match against, because the system will not produce the following paragraph, and was never meant to.

Most published guidance walks up to this and steps around it. It will tell you that AI systems in regulated work must be validated, that you should run IQ, OQ and PQ, and that audit trails need to satisfy 21 CFR Part 11. All true, and none of it answers the question a validation lead actually has to answer: what does a passing OQ test look like when there is no single correct output to compare against? That gap is where a lot of RAG deployments in quality functions are sitting right now, called validated, with test evidence that would not hold up under a determined inspector.

There is a better first question than whether the system hallucinates: what evidence demonstrates that this assistant consistently performs within its intended use? Hallucination detection is an engineering concern that sits underneath that question. Intended use, objective evidence and consistent performance are the terms an inspector reasons in, and they are where the validation argument has to start.

The FDA has already shown its hand

You do not have to speculate about how a regulator treats unreviewed generative output. On 2 April 2026 the FDA issued a warning letter to Purolea Cosmetics Lab. The firm had used AI agents to create drug product specifications, procedures, and master production and control records, and had put those records into use without confirming they were accurate and compliant with CGMP.

The finding sits squarely within the Quality Unit responsibilities already established by 21 CFR 211.22, which makes the Quality Unit responsible for approving or rejecting procedures and specifications that affect the identity, strength, quality and purity of the drug product. No new AI rule was needed. A regulation older than the technology was applied to a new drafting route, and it landed where you would expect. The model was never the issue. The finding rested on the absence of review and the absence of evidence that review had happened. When the firm explained that its AI agent had not identified a required process-validation step, the position was straightforward. Software can assist a task. It cannot absorb the firm's responsibility for performing it. Accountability did not move.

That is the frame I hold every RAG validation up against. The inspector will not ask whether your model is clever. They will ask what the output was, what it was checked against, who checked it, and where that evidence is now. The same accountability principle applies when a RAG assistant drafts a deviation summary, suggests a CAPA classification or answers a question about an SOP. The intended use, the criticality and the control design may differ from Purolea's, but the organisation still owns the resulting decision and the resulting record. Fluent, plausible output carries no evidentiary weight on its own. It carries weight only when the record can be traced back to an approved source and an accountable human decision.

The system under test is not the model

The most common mistake is to treat a RAG assistant as a language model with a wrapper, and to point validation at the model. A RAG assistant is an operating workflow of several parts, each of which fails in a different way:

• approved source content
• the retrieval mechanism
• the prompt instructions
• the language model
• the guardrails
• human review
• monitoring

An excellent model retrieving the wrong document still produces an unacceptable answer. Flawless retrieval cannot rescue an answer built on source content that should have been retired six months ago. The validated state belongs to the complete workflow, not to any single part of it. That is why a validation summary describing how well the model performed is answering the wrong question.

The output remains under test throughout. What changes is that it is no longer tested against one fixed string. It is tested against whether it stays inside a defined performance and safety boundary. Acceptance criteria move from exact text matching to measurable properties of the answer: required information is present, every claim is supported by approved evidence, defined prohibited behaviours are absent, and the system responds safely when the evidence is insufficient rather than guessing. Both halves matter. Negative criteria alone, the list of things the answer must never do, cannot be proven across an unbounded input space and were never sufficient on their own.

Related reading: Where AI sits in the quality system, and who ends up owning it once the implementation partner has gone: What Exactly Did Your Implementation Partner Leave Behind?

Qualify retrieval against a reference set

Retrieval is usually more directly measurable than generation. For a controlled set of queries, the validation team can define the approved documents or passages a competent user would be expected to find, then measure whether the retrieval layer returns them within an acceptable rank.

I choose the word measurable deliberately, and I avoid the word deterministic. Modern retrieval uses semantic embeddings, approximate nearest-neighbour search, hybrid ranking, rerankers and metadata filters. It can be repeatable under a fixed configuration while falling short of true determinism, and there is rarely a single indisputably correct passage, since several approved sources may be valid. So I qualify against a reference set and against metrics rather than against one right answer. The practical measures are familiar to anyone who has evaluated search: recall at k, precision at k, mean reciprocal rank, citation coverage, and the one that matters most in a regulated corpus, the obsolete-document retrieval rate. When SOP-114 is superseded, does the assistant stop serving it the moment it is retired, or does it keep citing a withdrawn document in fluent, confident prose? That has a right answer, it is a pass or a fail, and it is where the highest-risk failures actually live.

Retrieval built purely on semantic similarity has a specific failure mode worth naming, because it looks like precision and is not. A pure vector search can happily surface the passage on 21 CFR 211.68 when the query concerned a different, similarly worded clause from another section, because the two read as conceptually close even though they are legally distinct. A retrieval layer built for a regulated corpus needs keyword matching running alongside the semantic search, so exact citations and defined terms are preserved rather than only approximated.

One objection comes up in nearly every validation discussion, so it is worth answering directly. Setting the model's temperature to zero is supposed to force identical output for identical input, and teams often plan their OQ around that promise. It does not reliably hold. Inference on modern hardware batches requests together for throughput, and the floating-point arithmetic behind that batching does not distribute identically across every batch composition, so two candidate outputs that are close in probability can still land differently between runs. Temperature zero narrows the variation. It does not remove it. A test script written on the assumption that it does will fail intermittently for reasons that have nothing to do with whether the retrieved answer was actually right, which is a frustrating way to discover the assumption was wrong.

Qualify generation against defined properties

Generation is bounded rather than fixed, and the bound is what I qualify. The claim under test is narrower than a specific paragraph the model is expected to produce: whatever it produces has to hold defined properties, grounded only in the retrieved approved sources, complete enough for the task, free of unsupported conclusions, correctly expressing uncertainty, and refusing or routing questions that fall outside its approved knowledge boundary. Each of those is testable on output whose exact wording changes every run.

Some of this is better engineered in than tested for afterwards. Where the assistant has to produce a structured field, a document reference, a classification, a figure pulled from a batch record, that field can be constrained at generation time to a defined schema, so the model is physically prevented from emitting a value outside the permitted format or vocabulary. A separate, ordinary piece of deterministic code can then check that any number appearing in the answer also appears, verbatim, in the retrieved source text, before a reviewer ever sees it. Qualification still has to cover the rest of the answer. It no longer has to cover a category of failure that the architecture has already closed off.

This is where the title stops being a figure of speech. Because the system does not repeat itself, a single successful run proves almost nothing. High-risk scenarios have to be executed repeatedly, with the distribution of results assessed against predefined criteria. A system that answers nine questions cleanly and invents a critical instruction on the tenth has not passed.

Its average answer looked good, and average is not the standard a batch record is held to.

The scenario library is the qualification, and it should be built to probe the operating limits rather than the easy path:

• straightforward retrieval
• conflicting source documents
• superseded or obsolete procedures
• incomplete records
• ambiguous questions
• requests outside the approved knowledge boundary

Two disciplines keep this honest. The challenge set must not be tuned repeatedly against the system until it passes; separate the development set from the final acceptance set, or you are marking your own homework. And the edge cases should be drawn from real operating conditions, including rare and adversarial inputs, not from the comfortable examples that were easy to imagine. The draft Annex 22 on AI in GMP formally excludes generative models from its scope, but its emphasis on independent test data and on preventing the development team from contaminating the acceptance set is sound design thinking that applies here directly.

Where the regulators are pointing

It is worth being precise about the European position, because it is easy to overstate. The draft Annex 22 on artificial intelligence in GMP is still a draft and is not law as of writing. It scopes its critical-application requirements to static, deterministic models, those that give the same output for the same input. Generative models and large language models fall outside that scope, and the draft says they should not be used in critical GMP applications. Where they are used in non-critical GMP applications, qualified and trained personnel should remain responsible for confirming that outputs are suitable for their intended use, with human-in-the-loop control.

Read as a design constraint rather than a permission slip, that tells you exactly what a RAG assistant has to be: bounded, supervised, and kept away from decisions it is not qualified to make on its own. Your validation exists to prove the bound and to prove the supervision.

This piece stays focused on the validation mechanics rather than the wider regulatory picture. For how Annex 22 sits against Annex 11 and what it changes for GMP more broadly, see Annex 11 before Annex 22.

PQ is not the finish line

Point-in-time qualification cannot sustain a system like this, and annual revalidation will notice a problem roughly nine months too late.

A RAG assistant qualified in January is not the same system in October, even if no one has touched your configuration. The provider may have updated the underlying model. The source corpus has grown, and retrieval behaves differently against a larger, messier index. Documents have been superseded. Prompts have been refined. None of these is a software defect, and that is precisely the danger, because a change-control process tuned to catch defects and configuration edits is blind to every one of them. Nothing breaks loudly. It drifts, without ever throwing an error or returning a blank field. Instead it keeps answering, fluently, and the answer stops matching the approved source somewhere along the way while the confident tone gives no sign that anything has changed.

So PQ cannot be the final evidence. The qualified baseline has to be followed by ongoing performance monitoring: a fixed regression set with known-good source mappings, re-run on a defined cadence and on every relevant model, prompt, retrieval or corpus change, with results trended and a threshold that triggers investigation before an analyst relies on a degraded answer. I treat this as ongoing process verification rather than a one-off qualification event, and that framing is the right instinct. You are sustaining a living system that keeps moving under you.

The supplier relationship is part of the validated state here, and it is an architecture decision, not a paperwork one. The arrangement has to establish whether model versions are fixed, how version changes are notified, which configuration is identifiable in the audit record, and what events trigger regression testing. If you cannot answer those four questions about a model you did not build, you do not yet control the system you have deployed.

One model, five layers

The whole argument reduces to a table a validation lead can put in front of an inspector:

One measure belongs on that human review row that rarely makes it into a validation protocol: how often the reviewer disagrees with the assistant. When a reviewer accepts the output every single time, that consistency is itself the finding, because it usually means the review has become a formality the record cannot tell apart from a genuine check. A documented rate of disagreement, however small, is what tells you a human is still doing the job the process assumes they are doing.

What the inspector leaves with

Everything above collapses into one question, and it is the Purolea question wearing different clothes. For any given answer the assistant produced, can you reconstruct what it was built from, what checked it, and who stood behind it?

Reconstruction means something specific, not a general assurance that records exist somewhere. For any answer, you should be able to pull the query that produced it, the exact source passages retrieved and their version identifiers, the output as generated, the review action taken against it, who took that action, and when. That level of detail is a logging requirement, built into the architecture from the first design decision rather than assembled in the days before an audit.

If the answer is a validation summary saying the assistant was tested and performed well, you have described a successful demonstration and nothing more durable than that. If the answer is retrieval qualified against an approved reference set, generation qualified against defined properties and executed repeatedly, a human review step whose evidence is retained rather than assumed, and a monitoring regime that proves the boundary held over time, then you have something that survives the meeting. The model does not have to be trustworthy for that to hold. You designed the system so that it never needed to be.

The systems I have seen pass are the ones designed for this from the first architecture decision, where the boundary, the review and the evidence were built in and did not have to be reconstructed under inspection pressure. So I will end on the one contrast this whole piece turns on.

A generative system in a regulated process is not validated by proving it is right. It is validated by proving it cannot quietly be wrong without someone accountable seeing it first.

This reflects how I approach RAG validation on live engagements, including a production complaint-intake pipeline built with human review designed into the record from the start. If you are putting a generative assistant anywhere near a GxP process and are not yet sure your test evidence would hold, that is the conversation to have before the inspection, not during it.

See how Navata validates AI for regulated environments →

More from Navata Insights → /insights

Views expressed are personal and do not represent any employer or client.