6 AI governance practices for clinical trials

Repeatability.  Governance is what turns artificial intelligence in clinical operations from an interesting output into defensible evidence. That was the clearest takeaway from the panel I moderated on Day 3 of TMF Week 2026, and it is the one I keep coming back to as teams plan their next AI pilots.

I was joined by three people who have been in the rooms where those decisions get made:

• Michael Zwetkow, Chief Innovation Officer at Montrium, who leads the product team and the development of Montrium's products and services
• David Ives, Senior Director of Clinical Operation Systems at Madrigal, sharing his personal views rather than his employer's
• Peter Smedegaard Andersen, Founding Partner and CEO of DAQUMA, which builds a document automation platform that combines classic machine learning with large language models (LLMs)
  

Over the session, one question kept resurfacing: how do you trust an AI-supported decision enough to defend it? The panelists' answers fell into six themes. 

Six practices for governing AI in clinical operations

1. Treat AI output as evidence, not just an answer

In the TMF world, documents are the proof that a trial was run correctly. The same principle applies to AI. You need a record of how a decision was made: what went in, which model ran, and who approved the result. Without that record, you cannot show an inspector why the AI's output should be trusted.

Peter framed the governance gap around three needs:

• Evidence
• Traceability
• Repeatability

Crucially, compliance has to hold over time. As he put it, the question is whether the outcome you receive "is in fact a compliant outcome... not just by leveraging what was made compliant six months ago, but that is also compliant today."

David added a second risk. The TMF is "this treasure trove of data," and if organizations don't govern how AI uses it, people will use it anyway. "If we don't provide that context, it's happening elsewhere," he said. His fix is to offer the capability and the framework together: "If we have that, why go anywhere else? It makes the governance infinitely easier."

2. Extend vendor oversight to your vendor's AI providers

"How do we assess their capabilities, their security controls, all those questions that have to be answered by us because, ultimately, our customers are going to ask us those questions as well." - Michael Zwetkow, Chief Innovation Officer, Montrium

Software vendors building AI features face a choice: train their own model or use a pretrained one from a large provider. Michael said that "one of the biggest areas that we found challenging is the reliance on third parties." Using those models brings the latest capabilities, but shifts the emphasis to supplier management: "How do we assess their capabilities, their security controls, all those questions that have to be answered by us because, ultimately, our customers are going to ask us those questions as well." He explained that Montrium chose pretrained models over building its own, and stayed within the Microsoft ecosystem because it had already assessed Microsoft's controls, policies and certifications.

When you embed a third-party model, you also inherit its update cycle. David noted that sponsors are "still struggling with this sort of concept of oversight of technology deployed outside their walls to begin with." His advice was to understand the data flow and keep oversight dynamic, "because it's not a one and done static thing." 

Cost came up too. David said "cost is an implication that we really haven't figured out," and asked whether AI usage will become a pass-through cost on clinical trials. Michael described routing simpler tasks to smaller, cheaper models rather than always using the newest one. Peter added that token-based pricing is still new, and that "you can control some of it at a framework level, but other controls has to be introduced at the individual level."

3. Name a human owner for every AI-enabled system

When I asked Michael where to start building an AI governance framework, his answer was simple: "it started with accountability." Using AI to write code or answer a ticket "doesn't absolve you of the responsibility to make sure the output is accurate."

From there, Montrium assigns two roles to every system with embedded AI:

Ownership is a gap the industry already feels: lack of a clear governance owner is one of the barriers flagged in Montrium's State of AI in Clinical Quality 2026 report.

Michael also flagged retirement as a knowledge risk. With so much value now sitting in prompts, the question is "how do we make sure that that information is not getting buried and lost in these tools."

David has already seen the accountability gap in practice. Someone told him AI had said their approach was the best way to do it. His point: accountability "doesn't go away because AI told you to do it that way." I'd add that we rarely know how the AI was prompted, and these tools can be very convincing at telling us what we want to hear.

4. Validate the use case and the data, not just the code

Traditional computer system validation (CSV) assumes locked code and binary test results. AI does not work that way, and the guidance is still catching up.

Michael sees the industry's shift toward computer software assurance (CSA), a risk-based approach the US Food and Drug Administration (FDA) set out in its final CSA guidance for medical device quality systems, as still applicable: "it's all about taking a risk based approach, identifying what is the specific context of use." What has changed most is the weight on test data. The challenge is "how do we create these datasets of truly representative data" that cover real-world scenarios. And since real-world TMF content is inconsistent, "in some cases, you need your test data to be messy."

Peter drew a line between types of AI. In his view, certain classification models can be shown to be deterministic in their output, while content generation from LLMs is hard to justify without human oversight, whatever the risk profile. DAQUMA is monitoring the European Commission's draft Annex 22 on AI, written for Good Manufacturing Practice (GMP), but Peter said that understanding how regulators decide the risk level of an AI application is still tricky.

David agreed that intake and classification should lean deterministic, but argued generation should not: the value is in new ideas. On forcing generative AI into classic validation, he was blunt: teams are “going to go crazy if you try to validate it in the traditional context.” Security and privacy should be assured as a baseline; beyond that, oversight should be continuous.

5. Replace one-time validation with continuous monitoring

If AI cannot be locked down like traditional software, confidence has to come from watching how it performs. David described the model he wants: "a continuous sampling of the outputs and automation so that you can say, my threshold is this and we're still within that based on this continuous assessment."

Michael explained how Montrium measures this for classification. Every time a user overrides or changes an AI suggestion, "we're interested in that." Tracking overrides against the model's confidence level shows where humans almost always step in. That tells the team whether to retrain the model, or to stop showing a suggestion at all when the information is not in the document. "You don't want it to guess because it might mislead the person."

6. Build privacy, explainability and trust in from the start

An audience member asked about the hardest challenge where patient privacy, the AI model life cycle and regulations such as the General Data Protection Regulation (GDPR) and the EU AI Act meet. Michael's starting point was the data flow: "So where is the data going?" Best practice is to strip personally identifiable information (PII) before data reaches a model, and to instruct third-party providers not to use your data to train future models.

Peter added two harder problems. First, making sure models never reproduce personal data about trial participants. Second, the EU AI Act's emphasis on explainability: "having explainability in all outcomes is also a very hard thing to prove continuously, and requires a fairly solid framework to do so." I'd note that even anonymized data can carry patterns that become re-identifiable in the right context, so the risk extends beyond your data processing agreement into model design and testing.

Another attendee asked how to measure whether AI is trustworthy. David's practical answer was to require references. An AI support agent should say "you should do it this way because SOP says this, job aid says this, work instruction says this." Michael tied trust back to monitoring: regulators will want to see continuous checking, and "taking the appropriate action when things deviate from your acceptable quality limits."

Where to start, based on your team

You don't need a full framework on day one. Pick the first move that fits your situation.

Building AI you can stand behind

We did not leave the session with every answer, and regulators are still writing the rules. But the conversation pointed to where to focus: human accountability, supplier oversight, representative test data and continuous monitoring. Governance is not what slows AI down in clinical operations. It is what lets you rely on it when an inspector asks how a decision was made.

The full session recording is available on demand from TMF Week 2026. AI-Assisted Indexing is coming soon to eTMF Connect, built with the same human oversight our panel discussed. Talk to our team to learn more.

Frequently asked questions about AI governance in clinical trials

What is AI governance in clinical operations?

It’s the set of structures that let an organization trust and defend what its AI produces. In a regulated setting, that means being able to trace how an AI-supported decision was made, so the output counts as evidence an inspector will accept.

Who is accountable when AI gets something wrong?

The person using the output. Using AI doesn’t remove the responsibility to check that the result is accurate. Teams can assign an AI process owner, who confirms the system is fit for its intended use, and an AI system owner, who manages the technical side.

Can AI be validated like traditional software?

Not entirely. Traditional validation assumes locked code and pass/fail tests. A risk-based approach focused on the context of use still applies, with more weight on test data that reflects messy, real-world TMF content. Content generation should always have a human reviewer.

How do you monitor AI once it’s in use?

Track how often users accept or override AI suggestions, by document type. Set acceptable quality limits for each use case and review results on a fixed schedule. When confidence is low, showing no suggestion is better than letting the AI guess.