The US Food and Drug Administration (FDA) is a federal regulatory agency charged with protecting public health by ensuring the safety and efficacy of drugs, biologics, medical devices, and other products.1 To this end, the FDA conducts inspections of clinical investigation sites as a component of the Bioresearch Monitoring Program. The purpose of Bioresearch Monitoring Program is to (1) protect the safety, welfare, and autonomy of human subjects; (2) ensure clinical trials produce scientifically valid results; and (3) enforce applicable FDA regulations.2 A clinical trial site may be chosen for inspection if study sponsors file a complaint, clinical research has been discontinued, or a clinical trial is of special interest to public health.3 Inspection sites may also be chosen to audit data that have been submitted to the FDA as part of regular reporting during clinical trials.3
During an inspection of a clinical investigation site, FDA inspectors review medical records, case-report forms (CRFs), drug and/or device logs, and communications between sponsors and clinical investigators. Inspectors may also conduct interviews of study subjects and personnel among other activities. The Clinical Investigator Inspection List (CLIIL) describes the results of these inspections using codes describing the reason for the inspection, the final outcome, and specific violations.4,5 If the violations found are of “regulatory significance,” a formal warning letter is issued. Violations that meet “regulatory significance” are serious enough that they “may lead to enforcement action if not promptly and adequately corrected.”3 Warning letters describe these serious violations in a narrative format.3,6 Both the CLIIL and FDA warning letters are available to the public on the FDA website.4,6
Previous retrospective studies have analyzed and described the contents of FDA warning letters and the CLIIL database using different methodologies. Several studies developed coding methods to quantify the contents of warning letters issued to clinical investigators. In these studies, deviations from the investigational plan, failure to report adverse events, and inadequate informed consent were the most common violations reported.7–9 The sole study analyzing the CLIIL also shows that deviation from the investigation plan, inadequate informed consent, and inaccurate records were the most common violations.10 Overall, analyses of warning letters and the CLIIL show improvement in the conduct of clinical trials. Specifically, there has been significant improvement in keeping records, obtaining informed consent, and maintaining drug logs.7,10 In spite of these improvements, violations described as “deviating from the investigation plan” have either become more frequent10 or have not improved.7 However, no study has described, in detail, the nature of violations that have been categorized as “deviation from investigational plan,”7 “failure to follow investigation plan,”10 or other similarly vague descriptors.
In the current study, we analyzed the contents of the CLIIL through 2014. To enhance CLIIL data, we used FDA warning letters to develop more detailed descriptors of violations reported at clinical investigation sites.
For the purposes of the current study, we evaluated the CLIIL database comparing the 2005–2009 and 2010–2014 timeframes. Additionally, in a separate analysis, we analyzed FDA-issued warning letters during the 2010–2014 timeframe. See the Figure.
The CLIIL contains the names, addresses, inspection dates, reasons for inspection, violations found during the inspection, and final results of the inspection.4,5 We downloaded the CLIIL as a tilde (~) delimited text file from the FDA website and converted the file to a spreadsheet for analysis.4 The CLIIL was downloaded on July 29, 2016.
The CLIIL uses 3 categories to describe the reason for initiating an inspection: data audit (DA), for cause (FC), and information gathering (IG). A DA inspection is described as “an inspection in which the focus is on verification of study data.” An FC inspection is described as “an inspection in which the focus is on the conduct of the study by the Clinical Investigator.” The IG code was neither defined by the FDA nor found in the CLIIL.
To describe violations, the CLIIL uses 21 categories of violations listed in Table 1.5 Finally, to describe the final response from the FDA, the CLIIL assigns codes listed in Supplemental Digital Content 1, Table S1, http://links.lww.com/AA/C148. The primary codes assigned to describe the final response from the FDA are (1) no action indicated (NAI), (2) voluntary action indicated (VAI), or official action indicated (OAI). An OAI response indicates that a serious violation has been found and must be remedied to continue the trial at that site.5 Other codes found in the CLIIL to describe final responses include cancel (CANC), reference (REF), and case closed with memo to file (MTF).
For access to FDA warning letters, we utilized the Inspections, Compliance, Enforcement, and Criminal Investigations database available on the FDA website.6 We queried warning letters related to clinical investigators by selecting the “Browse Warning Letters by Subject” option. We then selected the search criteria “Clinical Investigator” and “Clinical Investigator-Human Trials.” This search generated a list of warning letters that contain narrative descriptions of violations found. The letters also cite relevant FDA regulations from the Code of Federal Regulation.11 All warning letters were read from the FDA website between August 1 and August 12, 2016.
Analysis of CLIIL
For the purposes of analysis, we compared inspections from 2010 to 2014 to the previous 5 years (2005–2009). No inspections were excluded from analysis.
To quantify the reasons for inspection, we calculated the percentage of each reason for inspection by dividing the frequency of each reason for inspection (FC, DA, or IG) by the total number of inspections reported during the respective time period. Similarly, for quantifying the final result of each inspection, we divided the frequency of each inspection result (NAI, VAI, OAI, CANC, REF, MTF) by the total number of inspections during the respective time period. As done in previous studies, we combined all OAI and VAI codes for comparison because the use of some codes is inconsistent over time.10 To analyze the frequency of each violation, we divided the frequency of each violation by the total number of violation codes reported during the respective time period.
Coding of FDA Warning Letters
After letters were populated from the FDA website as described above, the authors independently read each letter issued between January 1, 2010, and December 31, 2014. We each then created a list of common violations. After collaboration, we (C.R. and S.N.) developed a list of 31 violations (Table 2). After we developed numerical codes for common violations, we (C.R. and S.N.) independently coded each warning letter to ensure consistency in the coding methodology. If there were any discrepancies, the authors collaborated to reconcile any differences in the coding with a moderator. We then quantified the frequency of the 31 violation codes.
Using descriptive statistics, we characterized the average number of inspections per year, proportion of reason for inspection category (DA, FC, or IG), proportion of each violation code (0–21), and proportion of each final result code (NAI, VAI, OAI, CANC, REF, MTF). We compared the 2010–2014 timeframe to the previous 5 years (2005–2009) using χ2 analysis. We did not exclude any inspections reported in the CLIIL. Sample size was determined to detect a 5% change in violation code frequency relative to all violation codes and was based on the percentage of the violation code “failure to follow investigational plan” (30.4%) previously reported in Morgan-Linnell et al10 for the 2000–2009 timeframe. This code was chosen to determine sample size because it was the most common violation reported and was shown to be increasing in previous decades. Thus, this measure was the center of our analysis. Based on a 2-sided α of .05 and type II error of 15%, 1585 inspections were required in each group to achieve adequate power. With an average of approximately 363.7 inspections per year in the CLIIL database from 2005 to 2014, a 5-year sample for each group was appropriate to achieve adequate power. The CLIIL was analyzed using SPSS v. 24 (IBM Corp, Armonk, NY).
We analyzed 3637 inspections occurring between January 1, 2005, and December 31, 2014, from the CLIIL. We also separately analyzed 60 FDA warning letters issued between 2010 and 2014. The inspections listed in the CLIIL did not necessarily match the FDA warning letters analyzed in the study because the warning letters are issued after the inspection is complete, and recent investigations are ongoing.
Analysis of CLIIL
The mean ± SD number of inspections each year between 2005 and 2009 was 375.2 ± 35 inspections and 352.2 ± 57 inspections between 2010 and 2014. There was no statistically significant difference between the average number of inspections per year between the 2 timeframes (P = .472). There were also no statistically significant changes in the reason for initiating inspections (FC, DA, IG) between the study periods (Supplemental Digital Content 2, Appendix, http://links.lww.com/AA/C149). For results analyzing the final result of each inspection between each timeframe (NAI, VAI, OAI, CANC, REF, MTF), see Supplemental Digital Content 1–2, Table S1, http://links.lww.com/AA/C148, Appendix, http://links.lww.com/AA/C149.
Analyzing the violations reported in the CLIIL between 2005 and 2009, there were 2147 individual violations reported from 1876 inspections. Between 2010 and 2014, there were 2141 individual violation codes reported from 1761 inspections. Several inspections revealed multiple violations. Eleven of the 21 violation codes showed statistically significant differences (P < .05) when comparing timeframes (Table 1). Only the percentage of “no deficiency noted” relative to all reported codes increased significantly comparing the 2005–2009 and 2010–2014 timeframes (16.86%–33.86%, respectively; P < .0001).
There was a statistically significant decrease in the proportion of total violation codes listed in Table 1, including “failure to maintain and/or document subject consent,” “inadequate drug accountability,” “failure to follow investigational plan,” “inadequate or inaccurate records,” and “failure to notify IRB of changes.”
Coding of Warning Letters
The frequencies of each code and percentage relative to all violations coded are listed in Table 2. Each warning letter coded had an average of 5.1 ± 2.1 violations. The maximum number of violations was 11. One investigator had 0 violations using the coding method used in the current study. The violation cited by the FDA in the letter and not accounted for in our coding method was related to state-specific regulations for reporting HIV cases.
Our analysis of the CLIIL shows significant improvement in the conduct of clinical trials comparing the 2005–2009 and 2010–2014 timeframes. In spite of these improvements, violations involving deviations from the investigational plan remain the most common violation code reported in the CLIIL through the 2010–2014 period (25.46%). In our analysis of FDA warning letters, we describe the specific nature of these violation codes reported in the CLIIL. Our coding of warning letters showed that the 5 most common violations were: not keeping adequate case histories for 2 years (10.82%), enrolling ineligible subjects (8.85%), failure to perform required tests and/or exams (8.52%), inadequate supervision or use of unqualified personnel (7.87%), and inconsistencies between medical records (7.54%).
In our analysis of the CLIIL, the improvement in the conduct of clinical trials was consistent in both our analysis of violation codes and inspection final result codes. Comparing violation codes reported in 2005–2009 to those in 2010–2014, there was an increase in the “no deficiency noted” code from 16.86% of all violation codes in the CLIIL to 41.80% while the total number of violation remained similar. Consistent with this finding, there was an increase in the proportion of NAI codes to the total number of final result codes (49.29% vs 55.15%, respectively). Commensurate with these increases, there was a decrease or no statistically significant change in all violation codes. The overall improvement in clinical trials with a significant proportion of violation related to failure to follow investigation plan was also found in a previous study.10 Interestingly, since the FDA began reporting their findings from inspections in the CLIIL in 1977, this code has increased substantially. This is possibly tied to the increasing complexity of clinical trials as emphasized in Morgan-Linnell et al.10
The overall improvement demonstrated in this study, we believe, is attributed to the collaborative effort of researchers, industry, nonprofit organizations, and regulators. The regulatory framework and guidelines that govern clinical research have changed significantly in the past decade. Unlike in previous decades, research is mostly driven by the equal partnership of academia, various foundations, industry, clinical investigators, and other stakeholders in the research process. This partnership has made the researchers more informed about the requirements for conducting research and also bought more transparency in the research settings. One of the newly introduced partners into the field of clinical research is Clinical Research Organizations (CROs); they have been a driving force in the improved execution of the major clinical trials.12 The partnership between large pharmaceutical companies and CROs has generated a paradigm shift in the clinical trial process where pharmaceutical companies mostly focus on developing drugs and CROs work to bring successful discoveries to market quickly and safely. As such, CROs do not have any interest in the outcome of the study and are equipped to carefully monitor the conduct of clinical trials.12 Additionally, implementation of new technological innovations such as risk-based and centralized monitoring have improved oversight of clinical trials.
Representative Examples of Violations in FDA Warning Letters
The following are 2 examples of violations found in our analysis of FDA warning letters. These examples demonstrate how using FDA warning letters provides enhanced detail of violations in a way that is instructive to clinical investigators and the research community. They also demonstrate how the information from FDA warning letters can better identify trends in the violations than is currently possible using the CLIIL. The following examples represent violations related to (1) not keeping adequate case histories for 2 years, and (2) inappropriately delegating responsibilities.
- In a study of a new investigational drug,13 the sponsor utilized electronic CRFs (eCRFs). After the study at this particular site concluded, it was chosen for inspection by FDA investigators. During the inspection, the clinical investigator could not produce all of the data collected during the course of the study. The clinical investigator states that all of the data were on a sponsor-provided laptop that was taken back by the sponsor’s monitor when the study ended. While the data were obtained from the sponsor, it is still the clinical investigator’s responsibility to retain records at his or her site for 2 years after the investigation concludes. This example is instructive for clinical investigators. With the use of eCRFs (versus paper records), it may not be obvious that these records need to be copied electronically for the clinical investigator’s own records. Moreover, this case demonstrates the need for more detailed descriptors of violations to specifically identify violations and analyze trends over time. For instance, with the increased use of eCRFs in clinical trials, more investigators might be failing to retain their own records. Using the descriptor of “inadequate or inaccurate records” currently utilized in the CLIIL, we would not capture this possible trend because the descriptor is too vague. By analyzing FDA warning letters as we have in the current study, we are able to capture the violations with sufficient detail to be informative and instructive.
- In another study of a new investigational drug,14 the warning letter revealed the investigator did not properly delegate responsibilities. Specifically, in the course of the investigation, it was found that surgical residents who were not listed in Form FDA 1572 had ordered study medication. According to the investigational plan, the drug should have only been ordered and administered by the primary investigator himself or under listed subinvestigator who is responsible to the primary investigator. In reading and coding an FDA warning letter, we have again identified a problem that could not be identified using the CLIIL descriptors “failure to follow investigational plan” or “failure to supervise or personally conduct the clinical investigation.” As many clinical investigators have ever-increasing responsibilities, especially in an academic setting, this inappropriate delegation of responsibilities and failure to personally conduct components of an investigation are likely common. With the more detailed descriptors developed in this study, the research community is able to more accurately identify and therefore target these kinds of violations for improvement measures.
Opportunity for Improvement of Clinical Trials
Understanding the nature of the violations described in FDA warning letters is critical for clinical investigators because they are ultimately responsible for conducting clinical trials ethically and in compliance with FDA regulations.15 By knowing how clinical investigators most commonly violate FDA regulations, future investigators can avoid these violations if they are inspected themselves. More importantly, understanding past violations will help clinical investigators to protect patients participating in clinical trials and the many more patients who will be affected by the clinical trial results.
To this end, there is a need for more granular data to more precisely assess the conduct of clinical trials and educate clinical investigators. The vagueness of the descriptors used in other studies and by the FDA is not adequate in analyzing clinical trials as they become more complex. More detailed descriptions of common violations are needed to help clinical investigators understand problems in the conduct of clinical trials. The significant benefit of this study is how we have enhanced the CLIIL data by reading and coding FDA warning letters. We have described several common violations in more granular detail and provided examples of violations that have never been described in the literature. This more detailed information will help investigators target specific problems in the conduct of clinical trials.
Based on the results of this analysis, we recommend that clinical investigators more carefully scrutinize inclusion and exclusion criteria. We also can recommend that investigators make sure that all follow-up tests are performed and evaluated. As noted above, use of CROs will be essential for ensuring that clinical investigators meet the growing demands of ever more complex clinical studies.
There are some limitations to this study. Concerning the CLIIL, we noticed that there are differences in the CLIIL database comparing the current analysis to what was reported in Morgan-Linnell et al.10 For instance, our analysis of the CLIIL (not included in this manuscript) shows that 9492 inspections occurred between 1977 and 2009, while Morgan-Linnell et al10 reported that 9481 inspections. Morgan-Linnell et al10also reported 372 OAI designations while we found 371. These slight differences in the data might represent an update to the CLIIL or differences in methodology. This represents a weakness in our study and possibly the CLIIL dataset. This highlights the importance of having reliable data and consistent analytical methods. It is critical to have accurate data and analysis because they allow investigators to accurately self-evaluate and identify problems in how clinical trials are being conducted. Another limitation of this study is that we were not able to capture every violation in our coding of the warning letters. A more comprehensive and granular analysis of warning letters is warranted. Additionally, it is possible that the interpretation and enforcement of regulations and how they are reported in the CLIIL have changed significantly over time. This would introduce unpredictable confounding variables into the data. Finally, it is important to note that in relation to violation frequency between different timeframes in the CLIIL, this study was not designed to statistically prove that any particular violation has increased or decreased between the 2 time points. Our study is more focused on the qualitative nature of the content than the quantitative results. Adjusting for multiple comparisons would otherwise be necessary. However, due to the exploratory (as opposed to confirmatory) nature of the study, we are reporting all our P values unadjusted for multiple comparisons. Thus, comparisons between timeframes should be interpreted with caution.
The legal and ethical conduct of clinical trials relies on clinical investigators. Part of ethical research is producing scientifically valid results. This is ensured by conducting studies according to the investigational plan so the results can be interpreted properly.15 While we have demonstrated an overall improvement in the conduct of clinical trials, “deviation from investigational plan” remains a significant proportion of violations. To better understand how clinical investigators are deviating from investigational plans, we qualitatively and quantitatively analyzed FDA warning letters revealing several common violations in sufficient detail to help clinical investigators avoid these violations in their own clinical trials. This study provides a model for describing violations with the detail that is needed to adequately inform clinical investigators and the research community as clinical trials become more complex.
Name: Christopher A. Romano, BS.
Contribution: This author helped with the conception and design of project, data collection, data analysis, drafting the article, critical revisions, and final approval of manuscript.
Name: Singh Nair, MD.
Contribution: This author helped with the conception and design of project, data collection, data analysis, critical revisions, and final approval of manuscript.
Name: Ellise S. Delphin, MD, MPH.
Contribution: This author helped with drafting the article, critical revisions, and final approval of manuscript.
This manuscript was handled by: Nancy Borkowski, DBA, CPA, FACHE, FHFMA.