A guide to improved TMF metrics: the backbone of your TMF health

Not having access to accurate, real-time TMF metrics throughout a clinical trial is like driving at night with the headlights off. Sponsors and CROs need to leverage these metrics to improve TMF management and remedy performance gaps. However, when we polled our audience in a recent webinar (Implementing Metrics and Completeness Reporting) on the accuracy of their TMF metrics, 53% of respondents answered “somewhat accurate”, 21% said “not accurate”, 21% answered "I don't know”, and just 6% said “very accurate”. Clearly, there’s work to be done!  
 
Measuring the most basic of study performance metrics is a frequently cited problem amongst clinical professionals, so let’s start by going back to the basics. Below, we’ll discuss what a TMF metrics program is, how defining a program will improve your TMF metrics, and how to begin implementing one at your organization (hint, it’s not a one-size-fits-all approach). Then, we’ll take a look at some of the core TMF metrics and share what value they bring to your clinical trial. 

The fundamentals of a TMF metrics program: 

By establishing clear expectations between your study partners, you’ll have a clearer idea of how to measure and benchmark your metrics during your clinical trial. The Metrics Champion Consortium writes that “defining artifact-level and TMF-level quality expectations are important for assessing quality as these expectations are used to generate different metrics.”   

It’s important to set up your expectations with defined processes and standards around the filing of clinical trial master file content for more accurate metrics, as all metrics have multiple dimensions including Program, Study, Country, Site, Process Zone, Time, etc. You can also use a form of eTMF planning as outlined by the 5 W’s: who, what, where, when, and why to outline these expectations more clearly.  

Having an electronic system that can report on these core TMF metrics in real-time will help you see whether you are staying inspection ready and applying ongoing quality control. This is especially true for scaling organizations, where resources may be tight and the workload may be heavier for smaller teams. But to learn from TMF metrics, they must be accurate, and a lot can go wrong if they aren’t (e.g. inspection findings, wasted resources, failing to reach your goals).  

The purpose of a TMF metrics program (4 key areas)

We hope you would already agree with us when we say that a solid TMF metrics program isn’t just important, but actually key to ensuring the calibre of your study output. To ensure that your TMF always contains all the essential documents, we must be able to measure it, which is why a TMF metrics program is always necessary to put in place. However, in case you’re still on the fence about just how significant metrics programs are, here are four crucial areas supported by implementing a TMF metrics program:

1. Compliance with TMF processes, TMF expectations, contracts and service level agreements 
2. Tracking health to verify trends and to be able to perform continuous improvements 
3. Planning of future studies, timelines, and resource requirements 
4. Performance of CROs/industry standards and internal functional lines  

The measurement of data in these areas can truly help us to better understand our TMF processes, as well as to identify the gaps and areas for improvement. We’re all looking for a complete TMF that tells the  story of our trial in its entirety, but we must balance this desire with limited resources and human errors. A TMF metrics program will support you in establishing an ongoing strategy that ensures the inspection readiness of your TMF. 

The benefits of using a TMF metrics program   

If you don’t know the gaps and the issues in your TMF, how can you begin to improve? How could you make your TMF the most reliable storyteller of your trial? And on an operational level, if you need more resources for managing your TMF, which data will support your request?

Here are six key benefits of implementing a TMF metrics program—the gift that keeps on giving:

1. It supports you in planning and realizing an effective strategy 
2. It helps you make fact-based decisions, the only kind that really matter 
3. It develops a common language through data 
4. It develops alignment, transparency and accountability across teams 
5. It enables continuous improvement, allowing you to build a higher quality TMF 
6. It improves inspection readiness 

Yes, we said inspection readiness - let’s elaborate!  

Since we’re required to be inspection ready at any time, we need to be able to measure our TMF health status at the drop of a hat. To do that, you need to ensure oversight and control of your TMF. This is where TMF metrics become your best friend.  
 
To be inspection ready, the completeness, quality and timeliness of your TMF must be guaranteed in compliance with the requirements—you can’t just wing it. To verify if the requirements are met, you must measure the data inside your TMF: whether all expected documents have been filed, whether these documents are accurate, and whether they have been filed in real time (rather than a month before inspection). Thanks to a TMF metrics program, it’s possible to implement corrective and preventive actions before an inspection occurs.  

The unique challenge of implementing a TMF metrics program   

A TMF metrics program is fundamental for your TMF management, but implementing one is no small feat. There’s no standard metrics program to use, no “one-size-fits-all” approach. Why? Metrics should be customized to meet your organization’s unique needs. It’s important to identify which parameter of your TMF you want to measure and when, because according to your answers, you’ll have different outputs. Different metrics can be run at different milestones. The frequency of evaluating metrics can vary according to different situations, so even if there is a reference model to look at—for example, the list of 21 metrics identified by the TMF Reference Model—the TMF metrics program is unique to your organization and the starting point for establishing it is to define what to measure and to what end.  

Your assessment and your answers will be the starting point for implementing metrics in your TMF.  

How to implement a TMF metrics program  

1. Determine the purpose of your metrics: Think about what you need to measure in your TMF, including completeness, quality, timeliness, documents volume, users, and more. Remember, TMF metrics are not a one-size-fits-all solution. You will see variation depending on your TMF format, such as whether you are working in a paper or electronic format.   

2. Identify your critical success factors (CSF): Ask yourself which elements are necessary for the success of your TMF. Think about the ultimate goal of your documentation processes in the context of both inspections and the trial as a whole. Personally, we’d be surprised if a complete, quality TMF wasn’t one of your primary objectives. Once you've determined which factors are crucial for the success of your operations, you can then build your metrics program around them.  

3. Identify your key performance questions (KPQs): These are the questions that metrics have to answer in order to determine if the process is meeting the CSFs. The answers to your KPQs are the key performance indicators (KPIs), which is a performance measurement that enables the evaluation of the success—or not—of a specific activity. Depending on the type of question, you will end up using different metrics. Let’s look at some examples: 

• • KPQ: Is the TMF set up in a timely manner? 

• • • KPI: The metric you would need to evaluate is timeliness.  

• • KPQ: Do the artifacts submitted to the TMF meet the quality standard? 

• • • KPI: In this case, the metric to track is quality.  

Identifying the purpose of your metrics, your CSFs, and your KPQs is only the beginning of implementing a TMF metrics program, though. Once you know what you need to track, you then have to figure out how to track it. This is where technology-empowered eTMF systems can be a lifesaver. For instance, Montrium’s eTMF Connect system provides access to powerful, accurate, real-time metrics that are displayed on a centralized dashboard, so you’re never in the dark as to whether or not you’re on track to achieve TMF success. 

The 3 core TMF metrics: Remember them, live them, breathe them. 

You probably already know that the “basic” or core metrics to track and measure in our electronic trial master file (eTMF) are completeness, timeliness, and quality. More advanced metrics can be determined between the sponsor and CRO according to what exactly should be measured for your specific project (e.g. cycle time, efficiency vs. cost). For scaling organizations, it’s crucial to get the hang of establishing metrics programs early on, so that you aren’t left scrambling as your team and workload grow. 

Let’s look at how to measure each of the big three in detail. 

Core metric #1: TMF completeness 

When it comes to measuring the completeness metric, what you want to ask is does the TMF contain all information to tell an accurate story of what happened in the study? Unfortunately, we’re not talking about a quick bedtime story, though. Your TMF will be considered complete when all the documentation collected throughout a clinical trial is available in the TMF in an organized and auditable way.  

Completeness metrics are calculated based on the status of artifacts; complete, expected, missing, lost, or revisited. Luckily, you don’t need to be a calculus wizard to measure completeness. The primary calculation that we see is percent complete, which can be determined using the following formula:  

Percentage complete = total number of final artifacts / total number of expected artifacts.  

This measurement of completeness is based on a predefined list of expected artifacts at the study, country, site and investigator level. As a result, completeness can be quite variable based on study design, geographic location, and other factors. 

Often, the problem is that completeness is maintained manually, and we rely on people’s knowledge of what’s occurred to provide this kind of information. This means that today, it can only be used as a very high-level indicator. Alas, there is a better way! We wrote a blog on TMF completeness reporting where you can find more information on how to gauge completeness with more accuracy when considering study events and study process information.   

Core metric #2: TMF Timeliness  

All eTMF systems are required to be maintained up to date throughout the study, and your inspector will want to see proof that the information provided was done so in a timely manner. So, like most things in life, it’s better not to leave everything until the last minute. 
  
The EMA’s TMF Guideline states that “the clinical trial master file shall at all times contain the essential documents relating to the clinical trial”. Similarly, the ICH E6's Guideline for Good Clinical Practice shares, "Filing essential documents at the investigator/institution and sponsor sites in a timely manner can greatly assist in the successful management of a trial by the investigator, sponsor and monitor.”  

That’s all fine and good, but how do we calculate timeliness today? Timeliness measurements include days to review, days to filing, percentage late filing, and issue resolution time. But the primary indicator we use to show the timeliness of your TMF process over time is the percentage filed on time. You will want the result of the first part of this formula to be less than the second, as this will tell you whether the artifact has been filed on time or not. You can calculate it using the following formula: 

% Filed on time = (final artifact date – artifact creation date) < expected days to filing / total final artifacts  

Timeliness is key when compiling the TMF as the longer it takes to get information into the TMF, the higher the risk that information may be inaccurate or lost. Think about retelling a story that happened a few years ago versus one that happened yesterday—in the first instance, some of the details are going to be fuzzy, or even downright incorrect. A TMF is no different, especially when you’re dealing with large studies with many different participants who are working within the study and who will need to provide content on time.   

Defining standards in your eTMF planning for how long it should take to get records into the TMF is key. Your team should also identify and clearly define when you consider something to be final. Then, you can work out what days make the most sense for you to be able to calculate timeliness (e.g. when the document was approved, filed, sent for submission). As always, preparation is key. 

Core metric #3: TMF Quality 

So, maybe your TMF has a lot of information in it, but how can you tell if it’s good information? How do you know if it’s accurate? These are the questions that the TMF quality metric seeks to answer. It’s often considered the most difficult to measure as it involves taking a deeper look at the content of the artifacts. In the case of a TMF, it’s quality and quantity that matter. 

The quality metric is a measure of whether document content, metadata, and indexing are complete and accurate. You may want to look at artifacts reviewed, ICF reconciliation, artifacts audited, rejection rates, anomaly rates, and risk scores when measuring quality. There are two standard metrics we typically see with regards to quality:  

% Reviewed = final artifacts reviewed (manual QC) / total final artifacts  

as well as  

Rejection Rate = total artifacts rejected / total final artifacts  

Quality is typically evaluated on an artifact-by-artifact basis through manual QC and review processes, but rarely do we focus on the larger picture of the TMF. To truly evaluate quality, we should also be looking more holistically at our ability to tell an accurate story across artifacts. You don't want to get sucked into one specific aspect of TMF quality and miss large-scale gaps and incoherencies. 

Consider how inspectors consume TMF content, where they’re looking at multiple artifacts that surround a specific process, event, or a period of time. Similarly, we should focus on logical quality checks across artifacts and not just on individual ones to make sure that the artifacts make sense collectively within the system. Risk-based metrics, to be extended by machine learning, would also greatly improve our ability to assess quality.   

The takeaway 

“Above all else, show the data,” said the renowned statistician Edward R. Tufte. Many have the impression that the TMF only belongs to the clinical team. However, documents are coming from many different cross-functional teams feeding the TMF with content. You can use metrics to motivate and spark up some good competition among teams and ensure the document content, metadata, and indexing are complete and accurate.  

By establishing a TMF metrics program and by having a clear grasp of the three core metrics over the course of your clinical trial, you will be able to better manage your TMF and vastly improve inspection readiness. And, in the case of tracking metrics, you can always work smarter—not harder—by investing in an eTMF system that will do the calculations for you. 

Metrics aren’t the only thing required for ironclad inspection readiness! Check out our Inspection Readiness Checklist below to find out more 👇