In the clinical trials arena, sponsors and CROs need ongoing access to accurate, real-time metrics to drive improvement in TMF Management and address any operational performance gaps. Metrics drive good behavior, and not taking the opportunity to learn from metrics can lead to a pattern of poor outcomes from clinical activities.
You’ve probably heard it before that the “basic” or core metrics to track and measure in our electronic trial master file (eTMF) are completeness, timeliness, and quality. Having an electronic system that can report on core metrics in real-time will help you see whether you are staying inspection ready and applying ongoing quality control. But to learn from metrics, they must be accurate, and a lot can go wrong if they aren’t (e.g. inspection findings, wasted resources, failing to reach your goals).
In a recent webinar we held on Implementing Metrics and Completeness Reporting, we took a poll where 53% of respondents answered their eTMF metrics were “somewhat accurate”, 21% said “not accurate”, 21% answered "don't know”, and just 6% said they “very accurate”. And in the TMF Fundamentals webinar earlier this year we held, most respondents shared that completeness and timeliness were equally the most difficult to track. This tells us that there is a great room for improvement in measuring and assessing eTMF metrics on an on-going basis to drive real-time inspection readiness.
As measuring the most basic of study performance metrics is a frequently cited problem amongst clinical professionals, we thought of going back to basics with this article to walk you through how to measure each of the core metrics and share what value they bring to your clinical trial.
Setting up clear expectations with eTMF planning
By establishing clear expectations between your study partners, you’ll have a clearer idea of how to measure and benchmark your metrics during your clinical trial. The Metrics Champion Consortium writes that “defining artifact-level and TMF-level quality expectations are important for assessing quality as these expectations are used to generate different metrics.”
It’s important to set up your expectations with defined processes and standards around the filing of clinical trial master file content for more accurate metrics, as all metrics have multiple dimensions including Program, Study, Country, Site, Process Zone, Time, etc. You can also use a form of eTMF planning as outlined by the 5 W’s: who, what, where, when, and why to outline these expectations more clearly.
Completeness, timeliness, and quality are metrics that life science organizations should measure to review areas of concern, whereas going deeper with advanced metrics may reveal the source of a certain problem. More advanced metrics can be determined between the sponsor and CRO according to what exactly should be measured for your specific project (e.g. cycle time, efficiency vs. cost).
Core metric #1: eTMF completeness
When it comes to measuring the completeness metric, what you want to ask is does the TMF contain all information to reconstruct and tell an accurate story of what happened in the study? Your TMF will be considered complete when all of the documentation collected throughout a clinical trial is available in the TMF in an organized and auditable way.
Completeness metrics are calculated based on the status of artifacts; complete, expected, missing, lost, or revisited. The primary calculation that we see is percent complete, which follows the following formula:
Percentage complete = total number of final artifacts / total number of expected artifacts.
This measurement of completeness is based on a predefined list of expected artifacts at the study, country, site and investigator level. As a result, completeness can be quite variable based on study design, geographic location, and other factors.
Often, the problem is that completeness is often maintained manually, and we rely on people’s knowledge of what’s occurred and to provide this kind of information. This means that today, completeness is only a very high-level indicator. We wrote a blog on TMF completeness reporting where you can find more information on how to gauge completeness with more accuracy when considering study events and study process information.
Core metric #2: eTMF Timeliness
All eTMF systems are required to be maintained up to date throughout the study, and your inspector will like to see proof that the information provided was done so in a timely manner.
The EMA’s TMF Guideline states that “the clinical trial master file shall at all times contain the essential documents relating to the clinical trial”. Similarly, the ICH E6's Guideline for Good Clinical Practice shares, "Filing essential documents at the investigator/institution and sponsor sites in a timely manner can greatly assist in the successful management of a trial by the investigator, sponsor and monitor.”
How do we calculate timeliness today? Timeliness measurements include days to review, days to filing, percentage late filing, and issue resolution time. But the primary indicator we use to show the timeliness of your TMF process over time is the percentage filed on time. You will want the result of the first part of this formula to be less than the second, as this will tell you whether the artifact has been filed on time or not.
% Filed on time = (final artifact date – artifact creation date) < expected days to filing / total final artifacts
Timeliness is key when compiling the TMF as the longer it takes to get information into the TMF, the higher the risk that information may be inaccurate or lost. Especially when you’re dealing with large studies with many different participants who are working within the study and who will need to provide content on time.
Defining standards in your eTMF planning for how long it should take to get records into the TMF is key. Your team should also identify and clearly define when you consider something to be final. Then, you can work out what days make the most sense for you to be able to calculate timeliness (e.g. when the document was approved, filed, sent for submission).
Core metric #3: eTMF Quality
Quality is often regarded as the most difficult metric to measure today as there are multiple metrics that allow us to evaluate the quality of the information stored in our TMF. Here we are trying to answer: is the TMF of enough quality to tell an accurate story? And take a deeper look at the content of the artifacts.
The quality metric is a measure of whether document content, metadata, and indexing are complete and accurate. You may want to look at artifacts reviewed, ICF reconciliation, artifacts audited, rejection rates, anomaly rates, and risk scores when measuring quality. There are two standard metrics we typically see with regards to quality:
% Reviewed = final artifacts reviewed (manual QC) / total final artifacts
As well as
Rejection Rate = total artifacts rejected / total final artifacts
Quality is typically evaluated on an artifact-by-artifact basis through manual QC and review processes. But rarely do we focus on the larger picture of the TMF. To truly evaluate quality, we should also be looking more holistically at our ability to tell an accurate story across artifacts.
Consider how inspectors consume TMF content, where they’re looking at multiple artifacts that surround a specific process, event, or a period of time. Similarly, we should focus on logical quality checks across artifacts and not just on individual ones to make sure that the artifacts make sense collectively within the system. Risk-based metrics, to be extended by machine learning, would also greatly improve our ability to assess quality.
“Above all else, show the data,” said the renowned statistician Edward R. Tufte. By having a clear grasp of these three core metrics over the course of your clinical trial, you will be able to better manage your trial master file and improve inspection readiness.
Many have the impression that the TMF only belongs to the clinical team. However, documents are coming from many different cross-functional teams feeding the TMF with content. You can use metrics to motivate and spark up some good competition among teams and ensure the document content, metadata, and indexing are complete and accurate.
With an advanced system like eTMF Connect to support trial processes, you’ll be able to use robust reporting features to accurately track metrics in real-time and show how you’ve achieved milestones and events to share the full TMF story.