Validating Your AI Was the Easy Part. What Happens After Creates Inspection Debt.
Pharma has spent decades learning how to validate computerised systems. The discipline itself has never said, "test it once and forget it." Change control, periodic review and maintaining the validated state have always mattered.
Yet many operating models still behave as though the difficult work ends when the validation report is approved.
AI exposes the weakness in that habit.
Even when the underlying model is frozen, the system around it rarely stays still. Prompts are tuned. Retrieval sources are updated. Reference material changes. Thresholds are adjusted. Vendors release new model versions. Each change can alter the output without looking significant enough, on its own, to trigger the controls normally associated with a major system change. None of it looks like a new system. It looks like ordinary maintenance, the kind nobody would think to route through change control. But the system an inspector looks at eighteen months after go-live is rarely the system that was validated at go-live, and the gap between those two things doesn't announce itself.
It shows up as a specific kind of failure. AI doesn't usually fail during the demo. It fails during reconstruction. The model performs well. The users trust it. Value is being generated. Then someone asks three simple questions: show me how this recommendation was produced, who approved this version, and how do you know the controls were still effective after it changed. Those aren't capability questions. They're reconstruction questions, and reconstruction is where a one-time validation event runs out of road.
FDA's warning letter to Purolea earlier this year is what that looks like in practice. The issue wasn't that the firm used AI to help draft GMP documents, it was that quality unit review didn't happen before those outputs were used, and a process validation requirement got missed because, in the company's own words, the AI tool never told them it applied. Nobody at that firm set out to skip review. The gap between what was assumed and what was actually happening had simply been allowed to grow, quietly, until an inspector found it.
None of this depends on Annex 22 being final. The EU AI Act is already in force and its requirements are being applied in stages. Draft Annex 22 adds another clear signal of where GMP expectations are heading. Waiting for every implementation detail to settle isn't a lifecycle strategy.
There's a name worth giving the gap this creates. Call it inspection debt: the accumulated gap between how an AI-enabled process actually operates and what the organisation can reconstruct, evidence and defend. Technical debt makes a system harder to change. Inspection debt makes a regulated decision harder to defend. It grows whenever the system changes faster than its evidence, controls and accountability, and it doesn't announce itself, until an inspector asks a question the organisation can't answer cleanly.
It tends to arrive through small, reasonable decisions rather than one bad one. A prompt gets adjusted without going through formal change control because it seemed too minor to bother with. A model gets new reference material, but nobody revisits the original risk assessment in light of it. Reviewers get comfortable enough with a model's track record that they quietly stop writing down why they agreed with it. Monitoring narrows to "is it still accurate" and drops "is it still under control." None of these, on their own, look like a problem. Together, they're exactly what an inspector finds.
A few signals are worth watching for specifically. Nobody can say with confidence which model version produced a recommendation made six months ago. Prompt changes are outpacing whatever change control process exists for them. Human oversight has become something everyone assumes is happening rather than something anyone can point to evidence of. Performance monitoring tracks accuracy but has quietly stopped tracking whether the governance controls around the model are still doing their job. Validation gets discussed as something that happened, past tense, rather than something the organisation still does. Any one of these is worth a conversation. Several together usually means the debt is already real.
One pattern shows up often enough to be worth naming directly. Organisations tend to split AI governance across teams that each own a piece of it, compliance holding the policy, IT holding the infrastructure, data science holding the model, quality holding the validation. Everyone is genuinely doing their job. During an inspection, though, those boundaries stop mattering. Inspectors aren't interested in which department owned which piece, they're testing whether the organisation can demonstrate control across the whole lifecycle, end to end. That's a different question to whether each piece individually looks fine.
Which is really the underlying shift. The question inspectors used to ask was did you validate the AI. What they're increasingly asking instead is how do you know it's still operating within the assumptions that made validation acceptable in the first place. The first question can be answered once. The second one can't. It requires evidence that keeps pace with the system, not evidence that was accurate on the day it was filed.
That's the real argument for treating AI governance as an operating discipline rather than a validation milestone. Validation proves a system was fit for purpose on a given day. Nothing about that proof survives contact with a system that keeps changing, unless something is deliberately built to keep it current.
The organisations that do well here won't necessarily be running the most capable models. They'll be the ones who can say, on any given day, which version was live, which risks were accepted, and what evidence backs the claim that it's still under control.
This analysis draws on FDA’s April 2026 warning letter to Purolea Cosmetics Lab — the first U.S. drug cGMP enforcement action citing AI overreliance — alongside the EU AI Act’s phased implementation and the EMA’s draft Annex 22 guidance, and practical experience governing validated Quality systems in regulated life sciences. Views expressed are personal and do not represent any employer or client.
