Navata
← All Library

Regulated AI

Deep

How Should Acceptance Criteria and Test Data Be Defined for AI in a GxP Workflow?

AI-assisted research and drafting · Practitioner reviewed by Rohith Karanam Sreedhar · 24 September 2026

ⓘ Library content is researched and drafted with AI assistance and reviewed by a Navata practitioner before publication. For original analysis and long-form practitioner perspectives, visit Navata Insights →

How to Define the Intended Use of AI in a GxP Workflow establishes what an intended-use statement must specify, and How to Classify the GxP Risk of an AI Use Case establishes how much assurance the use warrants. This reference covers the step between them and the test execution: converting a settled intended use and risk position into criteria that a test can actually pass or fail, and into a data set that makes passing mean something.

It exists because the prevailing pattern in the market is a headline figure. A validation plan states that the model must achieve ninety-five per cent accuracy, the test achieves ninety-six, and the plan is signed. Nothing in that sequence establishes that the five per cent which failed were not all the cases that matter, that the test data was not seen during development, or that ninety-five was ever the right number.

Nothing can be written until the reliance is settled

Acceptance criteria are a statement about how wrong the output is allowed to be before the workflow stops being defensible. That cannot be written before three things are fixed.

What the output is. A proposed value, a classification, a retrieved passage, a draft of text, a ranking, a flag for attention. Each has a different notion of correct.

What happens to the output next. Whether a person is required to verify it against source evidence before acting, whether it populates a controlled field directly, whether it changes a workflow state, whether it is advisory. This is the single most important input, because a downstream verification step changes what a residual error actually costs.

What a wrong output causes. Not in the abstract: which regulated decision becomes wrong, how quickly it would be noticed, and whether it can be corrected.

Teams that write criteria before settling these end up measuring what the model reports rather than what the process needs. The symptom is a validation plan whose acceptance criteria could have been written without knowing which workflow the model sits in.

Choose the measure that follows the direction of harm

Overall accuracy is the default and is usually wrong, because it weights every error equally and regulated workflows do not.

Take a classifier that proposes a severity for incoming deviation records. Two error types exist and they are not symmetric. Proposing a lower severity than warranted means a deviation may be investigated less thoroughly and escalated late. Proposing a higher severity than warranted means wasted investigation effort and some noise. An accuracy figure treats these as the same event. The acceptance criterion should not.

The general rule is to identify which error reaches a regulated decision undetected, and to make the criterion constrain that error specifically. In classification terms this usually means setting a criterion on sensitivity for the harmful class, or equivalently on the false negative rate, and treating performance on the other error as a separate, usually looser, criterion about operational viability rather than about safety.

Three practical consequences follow.

State both error directions, with separate thresholds. A single combined figure conceals the trade-off that the model developer will otherwise make on the organisation's behalf.

Where the model outputs a score with a configurable cut-off, the cut-off is a controlled configuration item and belongs in the change control scope, because moving it changes performance without changing the model.

Where the output is a retrieval rather than a decision, the harmful failure is usually a silent omission rather than a wrong answer, so the criterion should cover whether the material item was retrieved at all, not only whether what was retrieved was relevant.

Source the threshold from something you can defend

The number itself has to come from somewhere other than convention. There are three defensible sources and it is worth being explicit about which one is in use.

The process being replaced. Where the AI takes over a task a person or an existing rule performed, the most defensible threshold is the performance of that existing process. The European Commission's July 2025 consultation draft of Annex 22 takes this position for its scope, indicating that where a model replaces an existing process the acceptance criteria should be at least as high as that process's performance. That is draft material issued for consultation and is not currently an operative EU GMP requirement, but the logic holds independently of its status: accepting a replacement that performs worse than what it replaced needs a justification that few organisations would want to write down. The practical difficulty is that the baseline is usually unmeasured, which means establishing it becomes part of the work rather than an inconvenience to be skipped.

The tolerance of the downstream control. Where a human reviewer or a second system check remains in the path, the question becomes what error rate that control can actually absorb. This is a real engineering question rather than a formality: a reviewer who must check every output can tolerate a higher model error rate than one who spot-checks, but only if the review is genuinely capable of detecting the error, which depends on whether the reviewer sees the source evidence. A criterion justified by downstream review should therefore be accompanied by evidence that the review works, which is the reviewer-interface testing territory already addressed in the risk classification reference.

Recorded practitioner judgement. Where neither of the above is available, the threshold is a judgement. Record it as one: who set it, on what reasoning, what alternatives were considered, and what would change it. This is Navata's recommendation rather than a stated regulatory expectation, and labelling it honestly is more defensible than dressing a judgement as a derived figure.

Set the criteria before the test runs, and record the date. Criteria settled after seeing results are not criteria. The draft Annex 22 makes the same point within its scope by expecting acceptance criteria to be set before testing.

Make the test data adequate, and independent in a way you can evidence

The test set is where most of the real assurance lives, and where most of the weakness is.

Representative of the real input population. Not of the data that was convenient to obtain. If the model will see records from three sites, four document formats and two languages, a test set drawn from one site in one format supports a claim about that one case. Describe the input space the model will actually encounter, then show how the test set covers it.

Large enough for the claims being made. Define a sample-size or statistical-adequacy rationale before testing, including sufficient volume in each subgroup you intend to make a statement about, so the measured figure is not an artefact of a handful of cases. A subgroup with eleven examples does not support a performance claim about that subgroup no matter what percentage the eleven produce.

Labelled against a stated reference standard. Every test set has a notion of the correct answer, and it comes from somewhere: an expert adjudication, an existing approved record, a rule, a consensus of reviewers. Name it, describe how disagreements were resolved, and be honest about its own error rate. The credibility of the measured performance is bounded by the quality of the reference standard used to judge it. Where the reference standard is a single person's opinion, the test primarily measures agreement with that person's judgement rather than establishing an independent truth.

Independent of model development. This is the requirement most often satisfied in name only. The draft Annex 22 expresses independence strongly within its scope, expecting test data to have had no influence on model development, selection, training or validation, and describing procedural and technical controls that separate the people and data involved in training from those involved in testing. Operationally, independence is broken when the held-out test set influences training, model selection, threshold tuning or iterative development, or when it materially overlaps the development data. Organisational separation between development and testing can strengthen the control, but the essential point is that the test set itself did not influence those decisions. Decide how independence will be maintained across versions before the first test, because it is very difficult to restore afterwards.

Held under control. The test set, its labels, its provenance and its version are validation evidence. They need the same retention and integrity treatment as any other test artefact, and re-running a qualification against a test set nobody can reproduce establishes nothing.

Do not let the average absorb the subgroups or the hard cases

A headline figure is an average over a population, and averages hide exactly the cases a regulated process cares about.

Split the input space into groups that are meaningful to the process, for example by site, by document type, by record category, by language, or by whichever dimension the process treats differently, and assess performance for each group separately. The draft Annex 22 takes this approach within its scope, expecting the input sample space to be divided into meaningful groups where applicable with performance assessed per group. Where a subgroup is too small to support a separate claim, that is itself a finding: the intended use should exclude it, or the evidence should be gathered before it is included.

Then deal with adverse and boundary cases explicitly. These are the inputs the model is most likely to handle badly and least likely to encounter in a randomly sampled test set: malformed records, near-duplicate documents, superseded versions, out-of-scope requests, ambiguous or contradictory source evidence, and inputs deliberately constructed to be misleading. They should be tested as a named set with their own expected behaviour, and the expected behaviour is frequently not a correct answer but a refusal, a flag or an escalation. An acceptance criterion that only measures the model's performance on typical inputs cannot support a claim about how the workflow behaves when something unusual arrives, and unusual arrivals are the ones that turn into deviations.

When the output is generative and there is no single right answer

Much of the above assumes a comparison against a known correct result. A generative output, a drafted summary, a proposed narrative, a rewritten section, does not have one, and the draft Annex 22 expressly treats generative models separately from the specified static deterministic models within its scope. That does not make acceptance criteria impossible; it changes what they measure.

Three approaches are workable and are usually combined.

Property-based criteria. Define properties every output must have, each individually checkable: every factual assertion is supported by a retrievable source in the provided context, no content is introduced that is absent from the source, required elements are present, prohibited content is absent, and the output stays within the defined scope of the task. These convert a subjective judgement into a set of binary checks that different assessors can apply consistently.

Structured human assessment against a rubric. A defined rubric, applied by named assessors to a defined sample, with inter-assessor agreement measured rather than assumed. The rubric and the agreement measurement are the evidence; a reviewer saying the outputs looked reasonable is not.

Criteria on the surrounding control rather than the output. Where the output is always verified by a person before it becomes a controlled record, part of the assurance legitimately shifts to that verification step, and the criteria attach to whether reviewers reliably detect seeded defects. The related question of verifying AI-drafted artefacts before they become validation evidence is covered in What Verification Is Needed Before Relying on AI-Drafted Test Scripts and Acceptance Criteria as GxP Validation Evidence?.

Whichever combination is used, expect variability between runs and decide in advance how it is handled: how many outputs are assessed, whether the same input is run more than once, and what consistency across repeated runs is required. A single well-chosen example proves nothing about a probabilistic system.

Connect qualification criteria to monitoring thresholds

The criteria used to accept the system and the thresholds used to watch it afterwards should be derived from the same reasoning, and they are frequently not.

Acceptance criteria answer whether the capability performed adequately on a controlled test set. Monitoring thresholds answer whether it is still performing adequately on live inputs where the reference standard is usually unavailable in real time. The bridge between them is deciding, at qualification, which observable operational signal stands in for the measure that cannot be observed directly: reviewer override and correction rates, escalation rates, disagreement between the model and a second check, or the outcome of periodic sampled re-adjudication.

Name that signal and its threshold at the same time as the acceptance criteria, and state what breaching it triggers. ICH Q9(R1) supports proportionate formality in how that is decided, and EU GMP Annex 11 in its operative 2011 revision expects periodic evaluation confirming a system remains in a valid state. The EMA's reflection paper on the use of artificial intelligence across the medicinal product lifecycle similarly emphasises a risk-based approach anchored in the specific context of use. Within its own scope of production and quality system software for medical devices, the FDA's Computer Software Assurance guidance supports assurance activities proportionate to intended use and risk; it is not a general pharmaceutical GMP requirement and should not be cited as one. What triggers revalidation after a breach is a separate decision and is not resolved here.

Sources