Contents
Step 2: assess Core vs Recommended documents
Because we are asked to develop and use a risk-based approach for performing TMF document QC and because not all TMF documents are essential to tell the story, a TMF document assessment must be done. This TMF document assessment helps to identify which documents carry the highest risks and may require more scrutiny in terms of QC or review.
In doing this kind of document assessment, different variables must be taken into consideration:
- If a document is core or recommended
- If the nature of the document has a direct effect on patient safety and/or data integrity
Core or Recommended
The first thing we need to evaluate is which documents are core and which ones are recommended. If a document is considered core, it means that it must be filed in the TMF, otherwise the inspector will not be able to reconstruct the story of the trial. This critical absence could have a negative impact on the final outcome of the inspection, or even the clinical trial itself. This evaluation should be performed and documented during the Master TMF Index customization.
It is crucial to understand which documents are necessary to reconstruct the story of the trial. Once this exercise has been performed on the Master TMF Index, we can then develop trial-specific templates or adapt the Master Index for a specific trial based on:
- The trial design (interventional vs non-intervention, device vs non-device, etc.)
- The operational model (outsourced vs insourced)
- Trial-specific SOPs (document output as process evidence)
- Geographic region-specific requirements (country-specific artifacts or requirements)
The CDISC TMF Reference Model identifies core and recommended documents, which is a good starting point. However, it should be noted that recommended documents may become core and vice versa based on the trial design.
When developing a plan for risk-based TMF oversight, you may leverage the core and recommended classifications to define different oversight strategies in conjunction with other risk evaluation factors.
Step 3: Evaluate impact on patient safety and data integrity
The next step is to perform an assessment on artifacts that could have an impact on patient safety and/or data integrity. We use the risk log developed in the first step to drive this exercise. For each artifact, an impact classification is assigned:

Some points to consider when performing an impact assessment include:
- This assessment could be done once for the organization in question and then applied across all clinical programs and studies.
- It should still be possible to make adjustments for specific study types or therapeutic areas if necessary.
- When identifying possible impacts, it is often useful to think about the processes that the artifacts play a role in.
- There could be multiple impacts for a specific artifact and therefore the most critical impact will determine the classification level assigned.
Step 4: Analyze non-compliance with TMF principles
The fourth assessment focuses on non-compliance with TMF principles (completeness, quality, and timeliness) which could impact risk. We should evaluate these principles and score them based on severity. We can apply a formula which provides an overall average risk score for a given artifact based on query rates, issue type and TMF dimensions (country, site, document owner, artifact type etc.).
The score is calculated as follows:
- We calculate a query rate risk score (QR) as outlined in table 1 for each issue type (I) per TMF dimension (D).
- We multiply each query rate risk score (QR) by the corresponding weighting (W) based on issue type (I) as outlined in Table 2 to calculate the weighted risk score (WR) for each issue type (I) and each dimension (D).
- We calculate the maximum weighted risk score (MR) by multiplying the highest query rate risk score (QR) by the highest weighting (W) for each issue type (I). In this example, it is always 12 as the highest query rate risk score is 4 and the highest weighting is 3.
- We then divide the weighted risk score (WR) by the maximum weighted risk score (MR) to obtain the dimension risk score % (DR) by dimension (D) and issue type (I).
- Finally we take each of the dimension risk scores (DR) and average them for the artifact in question to generate the artifact risk score (AR).
Query Rate Risk Score (QR)
When evaluating completeness, quality, and timeliness, the primary factor that we could leverage for risk assessment is query rate.
Query rate is defined as the % of artifacts that have valid queries raised on them. The higher the %, the higher the number of quality issues. Query rate can be calculated not only at the artifact level dimension (D) but also at the site, country, trial, organization, document owner, and process zone level dimensions. We assign a query rate risk score (QR) to each of these levels based on the % query rate, which is then used with a weighting coefficient (W) to determine the dimension risk score:

It may be of interest to further break down risk scores by issue type. For example, a missing artifact may present a higher risk than incorrect metadata. Proper evaluation should be performed to validate the need for such granularity. In our example, we apply a weighting to quality risk scores based on issue type. For instance:

Once we have defined the different query rate risk scores (QR) and weighting (W), we calculate the dimension risk score for each of the relevant dimensions for the artifact in question. The examples below show dimension risk scores for artifact and document owner levels for all monitoring visit reports (05.04.03) that are owned by John Doe.


Once we have calculated all of the various dimension risk scores based on the artifact in question—in this case a monitoring visit report artifact 05.04.03 in process zone 05 for United States site US-001 that is owned by John Doe who works for ABC CRO—we calculate the overall average risk score (ORS). The ORS is simply the average of all dimension scores for this specific artifact and will be used to determine the level of oversight required for this artifact.
We may also want to take into consideration any queries that have been raised on this specific artifact in relation to quality and timeliness. We could add an additional risk score into the mix to take this into account.

Queries are typically raised when artifacts are being processed in the system either at the time of indexing, during initial QC, or during secondary QC and periodic review. In the future, we may see more queries being detected programmatically through the use of AI and edit checks.
We could also leverage historical query data to provide a larger body of data for our risk algorithms to provide even greater predictability.
Step 5: consider additional risk factors
Other risk factors may also be taken into account when calculating a risk score. These factors could include:
- Document Lifecycle: Does the document go through a workflow approval into the eTMF, or is it created outside the eTMF and uploaded as final? When the document goes through a document workflow inside the system, it means that it is drafted, reviewed and approved; because of this process, the quality is probably higher than a document created on a desktop and signed in wet ink from an ALCOA++ standpoint.
- Stand-Alone vs Package: Are there cross-checks to be done on this document? If the document is part of a larger group of artifacts required to describe a specific process or event, such as an ethics submissions and approval process following a protocol amendment, there could be many cross-checks that are required. On the contrary, if it is a stand-alone document such as an audit certificate, cross checks are unlikely to be required.
- Document Translation: Does the document need a formal translation? If so, we may need to do additional verification to ensure that the translation exists and is compliant.
- Document Location/Primary Source: eTMF is not the only system where TMF artifacts are stored—they can also be stored within other clinical systems with TMF signposts indicating where these records reside. These systems may not fall under the same TMF management team and therefore may represent additional risk from a TMF management standpoint.
- SOP Evidence: Most artifacts are governed by formal processes which are described in SOPs or trial plans. The procedural documents tend to dictate the documentation that must be produced, as well the content—and in some cases the format—of this documentation. In the case that an artifact is not governed by formal procedures, additional oversight may be required.
- Subject Recruitment Levels: Higher recruitment levels at clinical sites may require more oversight and present a higher risk of inspection.
- Key Trial Events: If specific key or higher-risk events occur, such as interim analysis, IP expiry extension, etc., then additional oversight may be required.
- Quality Events: Protocol deviations, procedural non-conformances, and any other trial-specific quality issues could increase risk factors.
- Notes to File: The existence of note to files may increase risk.
- Audit Findings: Any audit findings should also be evaluated and may increase risk scores.
- Trial Design Parameters: Certain aspects of trial design may increase risk, such as double-blind studies.
- Previous Performance: Evaluating previous performance in relation to completeness, quality, and timeliness for sites, countries, organizations and individuals could have an impact on risk scores.
These factors could generate additional scores which can be blended into the overall average risk score. Manually, it would be very laborious to use so many different factors; however, with technology it is possible to apply algorithms that will allow us to integrate these other factors.
As we develop more robust and technology-driven risk frameworks, and as data standardization and interoperability improves (i.e. DDF), we will be able to take into account a growing number factors to fine tune risk scoring. In the meantime, we need to find practical ways to highlight specific factors or events that allow us to adapt our RBA to consider the most important factors that can impact our ability to tell an accurate and complete story.
Using risk to optimize the quality check process
To optimize the quality check process, two distinct but impactful approaches can be followed: one focusing on a second level of risk-based QC with the assumption of a preliminary check on 100% of documents, and the other applying a risk-based approach at the first level of QC.

An RBA applied to the second level of QC:
Traditionally, the TMF quality check is composed of different levels of quality checks. The first level is done on every single document, with a focus of verifying compliance with ALCOA++ principles and GDocP. Once this exhaustive initial check is complete, a second level of QC, using RBA, is applied. This second level of QC is more focused on high-risk artifacts and aims to verify the links between TMF documents that enable the reconstruction of the entire story of the clinical trial.
While this approach can provide a comprehensive understanding of TMF inspection readiness, it is challenging—or sometimes impossible—to implement universally due to the large volume and diversity of checks that need to be undertaken. This is when a RBA can help us hone in on where the risks are highest or where issues may exist, and focus our secondary QC efforts in those areas.
The First level risk-based QC: Strategic and proactive
An alternative approach involves implementing a risk-based strategy at the first level of QC. Here, not every document undergoes the same level of scrutiny. Instead, documents are triaged based on their perceived risk. We consider factors such as criticality, impact on patient safety, and regulatory significance, and select artifacts that present the highest risks in these areas for QC rather than arbitrarily performing QC on 100% of all artifacts.
This approach takes into account the fact that not all artifacts are created equal, and that the QC process must be adapted to focus on higher-risk aspects of the trial.
When implementing such an approach, it is also important to perform periodic reviews to ensure that the risk-based approach is being applied adequately. One methodology for doing this would be to take a periodic sample of artifacts that were excluded from primary QC and perform a review.
If a significant number of errors are observed or artifacts were incorrectly excluded and present a higher level of risk than originally thought, then the risk-based approach should be adjusted.
Selecting the right approach
Choosing between these approaches requires careful consideration of trial specifics, timelines, and available resources. Some critical factors to consider include:
- Trial Complexity: High-risk trials may benefit from a more comprehensive initial QC.
- Resource Availability: Limited resources may steer teams towards a more targeted risk-based approach from the outset (first level of QC).
- Regulatory Landscape: Adherence to specific regulatory requirements may influence the chosen strategy.
- GDocP Knowledge: QC applied only to a portion of TMF documents can be more effective if we can ensure that document owners are trained and prepared on what ALCOA++ principles are.
- Lesson Learned: The experience with past TMFs can teach us that two levels of QC are better than one.
- Internal Processes: Company SOPs and WIs can be robust enough to support the approach of a second level of QC based on RBA.
- Quality Trends: Regular assessment of TMF quality issues can lead to a more valuable second-level RBA to QC (typing errors are not comparable to patient identifier errors).
How to leverage risk scores for TMF oversight
There are many paths that one can take to apply a risk-based approach. Using the methodology that we have outlined, we can then apply risk categorization and scores to define the amount of oversight that is necessary.
The first step is to evaluate the trial risk and impact on the TMF management process. The second step makes a distinction between core and recommended artifacts, and we define different oversight strategies for these two categories. The third step is to look at the risk classification in relation to an artifact’s impact on patient safety or data integrity. This is applied across studies. This classification combined with the average risk score calculated in step four and additional risk factors identified in step five will determine whether we only do an initial (ALCOA++) QC, or secondary QC and an additional periodic review. Finally, if historical data is available, we could also apply historical risk scores to specific artifacts to define whether a primary QC needs to be done at all. The two tables below outline sample risk-based oversight strategies for core and recommended documents.


We could further introduce methodology whereby if we see an increase in error rate, we could then adjust the rules to increase the amount of oversight for specific artifact types. Whichever strategy we implement, it is imperative to properly validate it to ensure that it is adequate. Moreover, we must continuously monitor the outcomes of the RBA as the trial progresses and make adjustments where necessary.
Remember, not all studies are created equal, and therefore you will need to take a flexible approach to risk management within the TMF.
Conclusion
We have seen from the five-step approach outlined in this paper that taking a risk-based approach to TMF management and oversight is complex. It involves many different factors and large volumes of data and information that need to be analysed for our approach to be comprehensive. To perform this type of calculation manually is extremely challenging, which is why more simplistic, less comprehensive approaches have traditionally been taken—until now.
As eTMF solutions’ and clinical systems’ interoperability evolves and improves, we will start to be able to apply more robust approaches. Standardization of TMF-relevant information will also help in applying a system-based approach to risk management. Human oversight is and will remain important to ensure that the risk-based model that we implement is valid and remains so.
As we automate risk management more and more moving forward, we will be able to start better leveraging the value of our TMF management teams. These teams will be able to leverage the risk scoring information and risk management strategies to provide more focused, relevant oversight to our TMFs. In turn, this will have a positive impact on the completeness and quality of information contained within our TMFs, and our ability to tell a complete and accurate story of what occurred in a trial during inspections.
References
FDA (U.S. Food and Drug Administration):
FDA Guidance for Industry - "Oversight of Clinical Investigations – A Risk-Based Approach to Monitoring" (August 2013).
FDA Guidance for Industry - "E8(R1) General Considerations for Clinical Studies" (November 2020).
FDA Guidance for Industry - "E6(R2) Good Clinical Practice: Integrated Addendum to ICH E6(R1)" (March 2018).
EMA (European Medicines Agency):
EMA Reflection Paper - "Risk-Based Quality Management in Clinical Trials" (November 2011).
EMA Guideline - "Good Clinical Practice: ICH E6(R2) Addendum" (November 2017).
MHRA (Medicines and Healthcare Products Regulatory Agency, UK):
MHRA Guidance - "Risk-adapted Approaches to the Management of Clinical Trials of Investigational Medicinal Products" (April 2011).
ICH (International Council for Harmonisation):
ICH Guideline - "E6(R2) Good Clinical Practice: Integrated Addendum to ICH E6(R1)" (November 2016).
WHO (World Health Organization):
WHO Guideline - "Handbook for Good Clinical Research Practice" (2016).



