Kitterman, Darlene R. MBA; Cheng, Steven K. PhD; Dilts, David M. PhD, MBA; Orwoll, Eric S. MD
Academic medical centers (AMCs), or institutions with the core mission of conducting clinical research, play a vital role in the discovery of new knowledge, the evaluation of current therapies, and the education of the medical and local communities.1 One critical role of an AMC is to conduct research that advances basic scientific observations to applications in medical practice. The National Institutes of Health (NIH) has made translational research a priority, in part through the formation in 2006 of the Clinical and Translational Science Awards (CTSA) program. In that effort to improve bidirectional translational science, four strategic goals were formulated—the first of which is to enhance the national capability for clinical and translational research.2 This goal includes increasing efficiency, quality, and safety in the conduct of research.3 As one of the first CTSA-funded sites, Oregon Health & Science University (OHSU) has undertaken efforts to enhance translational science by identifying and overcoming local barriers to clinical research. One such important barrier is enrolling sufficient numbers of participants in clinical studies to support their stated scientific objectives.
The recognition of barriers to successful participant enrollment in clinical studies is not new—a study over 25 years ago found that one-third of a cohort of 41 randomized clinical studies at the NIH recruited less than 75% of their planned enrollment goals.4 A sampling of 13 studies sponsored by the National Heart, Lung, and Blood Institute found that planned enrollment was completed for only two of the studies (15.4%),5 and more recent investigations of federally sponsored oncology trials show that the proportion of studies that failed to achieve minimum recruitment goals ranged from over 25% to nearly 40%.6–8 Furthermore, specific milestones can be used early in the enrollment period to predict eventual enrollment success for oncology studies.9 Studies that do not achieve planned enrollments are unable to support their intended scientific hypotheses, thus reducing their scientific relevance and the efficiency of the entire clinical research enterprise.
To determine the institutional scope of this problem, to understand its impact on resource utilization, and to set the stage for effective remediation, we analyzed the prevalence of studies that were unsuccessful in enrolling participants and their associated administrative costs at a single AMC during a four-year period. We discuss the economic implications of such studies and present strategies for addressing this issue.
Categorizing low-enrolling studies
We included all clinical studies at OHSU terminated between July 1, 2005 and June 30, 2009 (FY2006–FY2009) in this analysis. We defined terminated studies as those in which participant recruitment and all research activities, including participant follow-up or data analysis, were completed during the study period. We collected data from electronic records maintained by the OHSU institutional review board (IRB). This database includes all OHSU human studies and study enrollment information at the time of each continuing review and at study termination. We defined low-enrolling studies as those with zero or one participant enrolled at the time of study termination. We chose this definition of low enrollment based on a threshold that would be considered a failure to meet scientific end-point objects in almost every context. We included only terminated studies in the analysis, so we analyzed only confirmed final enrollment results. To better understand potential identifying characteristics of these studies, we examined low enrollment by (1) IRB review type, (2) academic unit conducting the research, and (3) funding mechanism.
IRB review type.
We categorized clinical studies by IRB review type: full-board, expedited, and exempt. Studies requiring full-board review, as defined within the Code of Federal Regulations on IRB functions and operations,10 were approved at a convened meeting with a committee quorum. Expedited or exempt studies were reviewed and approved by the IRB chair or other qualified reviewers without involvement of the board. For the purpose of this analysis, we compared the prevalence of low-enrolling studies with full-board review versus those with expedited/exempt review. We calculated the economic resources attributable to low-enrolling studies involving full-board review.
Academic units conducting the research.
We also assessed the prevalence of low-enrolling studies across academic units, including departments, divisions, or other organized research units. We limited analyses of academic units to those with more than 20 terminated studies between FY2006 and FY2009, which was the minimum sample size required to calculate median costs and avoid aberrant results.
Funding sources for clinical research include industry, government, and other sources. Industry-funded studies receive primary financial support from private corporations and include both those designed by industry and those developed by investigators at OHSU but supported by industry; the latter studies are typically termed investigator initiated. Government funding includes federal- and state-supported studies. The other sources category includes those studies sponsored by agencies (such as foundations) or other institutions, those supported by multiple sources of funding, and those conducted without any extra institutional financial support.
Calculating uncompensated costs due to low enrollment
We estimated uncompensated costs associated with low-enrolling, full-board-reviewed studies for FY2009 across four primary sources of administrative support for the development and conduct of clinical research: (1) the principal investigators (PIs) and their academic units, (2) the Clinical Trials Office (CTO) contracting unit, (3) the OSHU IRB, and (4) Sponsored Projects Administration (SPA).
PIs and their academic units.
This source of administrative support provides substantial upfront investment in the development of a clinical study. Each academic unit requires specific and, to some extent, unique processes and reviews. For example, the OHSU Knight Cancer Institute, a National Cancer Institute-designated cancer center, provides a separate scientific review of all oncology clinical studies. The PI and his or her staff must also prepare submissions to the IRB, negotiate and prepare study budgets, and implement the study, including collecting regulatory documents, interfacing with infrastructure units such as pharmacy and nursing, participating in investigator meetings and monitor visits, and creating recruitment plans.
CTO contracting unit.
The CTO contracting unit is responsible for coordinating and negotiating contracts for industry-sponsored clinical studies. This negotiation process includes the review of confidential disclosure agreements (CDAs), clinical trial agreements (CTAs), subcontracts, and amendments.
The OHSU IRB reviews all research involving human subjects performed by OHSU faculty, research staff, and students. All open, nonexempt clinical studies are reviewed by the IRB at least annually, and any study design modifications or unanticipated problems are reviewed as necessary.
Sponsored Projects Administration (SPA).
SPA manages the financial aspects of all OHSU-sponsored projects. For each research award received by OHSU, the award account must be initiated, managed, and closed. Expenditures related to the conduct of clinical studies are managed through this office.
Calculating the cost of administrative support
We separated the costs of administrative support for study development into three types: (1) start-up, (2) maintenance, and (3) close-out costs (see Table 1). We included only full-board review studies in the cost analysis because these types of studies are the most resource intensive. We only included costs attributed to low-enrolling studies that were not directly reimbursable.
Start-up costs are associated with activities required to prepare a clinical study for enrollment. Start-up costs include preparation of study materials, IRB initial review, preparation of study budgets, contract negotiation, awards setup, and study planning meetings. Costs associated with study team preparation at the academic unit level for new studies presented to the IRB in FY2009 are based on estimates of the average salary with fringe benefits for a study coordinator as well as previous estimations used by Durivage and Bridges11 and C-Change.12 The salary for study coordinators, including fringe benefits, is estimated to be $60/hour. These costs are an underestimation of the total start-up costs because they do not include the time and effort consumed by the study PI(s), biostatistician, or other study team members, such as coinvestigators and collaborators, pharmacists, nursing staff, clinical department staff, and/or academic unit administrative staff. Some academic units have centralized resources to negotiate budgets and prepare studies for IRB review, and OHSU runs a centralized office that all academic units can use for these activities. These centralized resources recoup their costs through a nonrefundable start-up fee payable on completion of the activities. However, many studies at our institution do not use these central offices and do not recover their start-up costs unless they enroll participants.
To estimate IRB personnel effort, we included the anticipated time required to support a full-board review and approval process. In calculating IRB cost estimates, we excluded industry-designed studies because the IRB charges review fees at initiation of those studies to recoup the cost of the review.
Industry sponsors often require CDAs before providing clinical trial documents. CDAs are reviewed by the CTO contracting unit to ensure that the terms and conditions are in accordance with OHSU policies. Before the initiation of industry-sponsored clinical studies, the CTO contracting unit negotiates the terms and conditions of the CTA with the industry sponsor. The costs associated with low-enrolling studies reflect the estimated percentage of low-enrolling studies across the total number of CTAs and CDAs negotiated. Additionally, award setup by SPA is required for all sponsored studies, and so the effort associated with setting up studies financially is attributed to all studies.
Maintenance costs are associated with the ongoing activities required to keep a study open regardless of whether participants are enrolled. For example, amendments to CTAs require review and negotiation by the CTO contracting unit, so costs are based on estimated efforts. Also, the IRB must review nonexempt studies annually, as well as any modifications to the study. In our analysis, we used the number of IRB modifications and annual continuing reviews reported by the IRB during FY2009 and determined uncompensated costs for conducting these reviews accordingly.
Close-out costs are generated when studies are terminated. Study termination must be reported to and reviewed by the IRB. In addition, SPA must close the study account, involving review and reconciliation of all study charges.
We summarized categorical and ordinal groups using univariate and cross-tabulated frequency distributions. We compared low-enrolling studies for each identified characteristic using the Kruskal-Wallis test. We calculated post hoc comparisons of statistically significant observations using the Mann-Whitney test. We maintained a maximum two-tailed alpha of .05 for determining statistical significance. We performed all statistical analyses with SPSS (version 17.0, Chicago, Illinois, www.spss.com).
The OHSU IRB determined that this study did not require approval because the data provided by the IRB did not include any identifiable information and were not obtained through interaction with human subjects.
In our analysis, we found a total of 837 terminated studies conducted by 57 academic units between FY2006 and FY2009, and we identified 260 (31.1%) as low-enrolling (see Table 2). This cohort of total terminated studies constituted 24.1% of all 3,470 clinical studies conducted at OHSU during the study period. Of the total sample of terminated studies, 383 (45.8%) required IRB full-board review, and, of these, 173 (45.2%) were low-enrolling studies. Of the 454 exempt or expedited studies (54.2% of all terminated studies), 87 (19.2%) resulted in low enrollment—significantly less than the 45.2% of full-board-reviewed studies (P ≤ .001). Full-board studies included those involving therapeutic interventions and were generally more complex than expedited or exempt studies.
Comparison by funding mechanism
Table 3 summarizes the low enrollment rates by funding mechanism (industry, government, or other sources). The other sources category (such as internal, foundation, and multiple sources of funding) was the most common source of financial support for the studies that we examined (416 of 837 studies; 49.7%). Studies supported by other funding mechanisms had the smallest proportion of low enrollment (85 of 416; 20.4%; P ≤ .001). Clinical studies that received funding from government sources (97 of 837; 11.6%) had the highest incidence of low enrollment (52 of 97; 53.6%). Across all review types, government-funded studies had a greater proportion of low-enrolling studies compared with industry-funded studies (P = .006). Of those studies requiring full-board review, government-funded studies also had the greatest incidence of low enrollment (36 of 48; 75.0%) compared with studies supported by industry (110 of 271; 40.6%) and other sources (27 of 64; 42.2%) (government versus industry: P ≤ .001; government versus other: P = .001). When comparing exempt/expedited studies by funding mechanisms, differences were observed between government (16 of 49; 32.7%) and other funding (58 of 352; 16.5%) (P = .006).
Comparison across academic units
We identified 16 of the 57 (28.1%) academic units in the sample with more than 20 studies. These academic units accounted for 603 studies (72.0% of the total terminated studies). The number of low-enrolling studies within these academic units ranged from 29 studies (10%) to 46 studies (67%).
Economic impact of low-enrolling studies
To understand the economic implications (uncompensated costs), we calculated the estimated unit and administrative costs attributable to the low-enrolling studies. We estimated the uncompensated costs associated with studies terminated with low enrollments for study start-up, maintenance, and close-out activities across the involved infrastructure (IRB, CTO contracting unit, and SPA). A complete list of identified costs and the variables used for the cost estimations are found in Table 1. Our estimated financial impact of low enrollment is conservative in that it does not include exempt or expedited studies (454 of 837; 54.2%) or PI time.
The total institutional uncompensated cost of studies that enrolled zero or one participant was approximately $990,000 in FY2009. Start-up and maintenance costs represented approximately $637,000 (64.4% of the total cost) and $350,000 (35.4% of the total cost), respectively (see Table 4). Of the start-up costs, the largest proportion was associated with study preparation (about $466,000 or 47.1% of the total cost), including developing study tools, and preinitiation and initiation meetings.12 We estimated the average cost to initiate one study to be $4,800. The largest subcomponent of maintenance costs was associated with study modifications (about $315,000 or 31.8% of the total cost). We estimated the resources allocated to IRB review of modifications and continuing reviews of low-enrolling studies to be $30,000 (3.1% of the total cost). We found close-out costs linked with low-enrolling studies (about $2,000 or 0.2% of the total cost) to be relatively small in proportion to both study start-up and annual maintenance costs.
Clinical and translational research in the United States has been characterized as fragmented, poorly coordinated, slow, and expensive.13–15 An important component of clinical research occurs at AMCs, but there are few objective assessments of its cost or effectiveness. Our findings focus on one major problem within the ailing system of clinical research: low-enrolling studies. Almost one out of every three terminated clinical studies at OSHU enrolled zero or one participant during the period of our research. Although our research was limited to a single AMC, it is highly likely that enrollment efforts and their costs are similar at other institutions; for example, research in oncology has shown a great deal of similarity in low-enrolling studies among different cancer centers.11 The costs of such low-enrolling studies are extensive and represent an important drain on research resources both locally and nationally, a particularly important issue in a constrained economic environment.
The financial implications of clinical studies that do not enroll enough participants are often overlooked, and, while being relatively opaque to individual researchers, such costs can quickly accumulate to represent a significant amount of an institution's resources. At our institution, the annual administrative cost for low-enrolling, full-board review studies was found to be almost $1 million for FY2009. Importantly, the consumption of these valuable resources presumably resulted in minimal scientific benefit. More worrisome, such studies indirectly may have prevented the conduct of other research that would have been more likely to achieve its primary end points through successful recruitment.
The scientific and financial implications of low enrollment warrant substantial efforts to foster awareness of their effects and to develop methods for prevention and mitigation. The largest proportion of cost was attributed to study start-up. Approaches to avoid this significant effort should include routine and objective feasibility assessments before supporting the initiation of a study. Ongoing maintenance costs were also substantial, suggesting a need for strategies to proactively recognize and close low-enrolling studies. Our research highlights three characteristics that might focus these efforts: (1) the type of IRB review required, (2) the funding mechanism, and (3) the type of academic unit conducting the research. Studies requiring full-board review and those funded by government sources more frequently involved low enrollment; hence, an earlier review of such potential studies at the concept level could be helpful. Preemptively preventing studies that may present potential barriers to feasibility will avoid the allocation of the critical and significant resources necessary to start and maintain a study that is at risk for low enrollment. As our data show, interventions to close studies once they exhibit symptoms of low enrollment are suboptimal because resources have already been spent and cannot be recovered. Finally, understanding and reducing the variation in study design and conduct among academic units could identify potential strategies directed to addressing low-enrolling studies. Our research highlights the overarching argument that implementation of improvements to clinical research cannot be done in isolation by any one group or one particular area of implementation; rather, there is a need to address barriers to clinical research from a concurrent and collaborative approach across all parties to create a more efficient and effective clinical research system, that is, taking more of a systems approach.15
In addition, two specific cultural issues must be addressed to reduce the proportion of low-enrolling studies: the paradox of abundance and sunk-cost bias. The paradox of abundance is the overestimation of institutional resources potentially available to support the nearly infinite number of possible clinical studies. Management literature shows that, with such a misperception, there is an increased time to market (in our case, achievement of recruitment goals), an increased number of unexciting products (or studies), and an increased number of failures.16 There must be a widespread understanding that all groups operate within the boundaries of constrained resources and that the commitment of those resources must be carefully undertaken early in the process of study inception. The other cultural issue, sunk-cost bias, in which the irrecoverable costs incurred in building the study hinder rational decision making in later phases, must be appreciated.17 It is well known that the level of effort and resources necessary to initiate clinical research is substantial; yet, inappropriately continuing to pursue those studies that have a high likelihood of low enrollment incurs additional financial and resource costs. Recognizing the point at which low enrollment will not be reversed, and discontinuing efforts on the study, would restrain cost accumulation.
Our study has a few limitations. First, our research underestimated the true costs of low enrollment. We included only studies involving full-board IRB review and used a conservative estimate that incorporated only the standard institutional costs common across all clinical research. We also chose to include only terminated studies in our analysis and did not account for those studies that are ongoing yet continue not to enroll participants. In considering the total level of effort and financial implications of low enrollment, it must be recognized that each academic unit has unique approaches to the development and conduct of clinical research that will variably affect study costs, which we did not take into consideration. Moreover, our assessment of economic implications of low enrollment did not consider activities such as preparing and writing study protocols by the PI, completing study-specific training, and conducting recruitment activities. Clinical and administrative activities, including drug dispensing and maintenance, nursing support, regulatory reporting, and screening of participants, were also not included in our analysis. Finally, our analysis did not account for intangible costs, including the detrimental impact of low-enrolling clinical studies to the reputation of the university and PI, and relationships built across the research community.
Improving the productivity of research is one of the fundamental challenges to transforming clinical and translational science and cultivating a culture of excellence across all AMCs. The etiology of low-enrolling studies must be further examined to determine the underlying causes and thus enable prevention and mitigation. As part of a comprehensive effort to reform clinical research on an institutional level, we are developing an objective understanding of the characteristics and impact of low-enrolling studies. Our findings set the stage for undertaking targeted approaches to reduce the problem and increase clinical research effectiveness.
The authors would like to thank the Oregon Clinical and Translational Research Institute, the OHSU institutional review board, and the Knight Cancer Institute for all their help and support for this study. Specifically, the authors wish to thank Susan Bankowski, JD, for providing most of the necessary data.
This study was made possible with support from the Oregon Clinical and Translational Research Institute; the National Center for Research Resources (grant number UL1 RR024140), a component of the NIH; and the NIH Roadmap for Medical Research.
The OSHU institutional review board determined that this study did not require approval.
The results of this study were presented in part at the 2010 Cancer Trial Accrual Symposium and the 2010 Annual CTSA Clinical Research Management Workshop, both held in Bethesda, Maryland.
1 Kassirer JP. Academic medical centers under siege. N Engl J Med. 1994;331:1370–1371.
3 Pober JS, Neuhauser CS, Pober JM. Obstacles facing translational research in academic medical centers. FASEB J. 2001;15:2303–2313.
4 Charlson ME, Horwitz RI. Applying results of randomised trials to clinical practice: Impact of losses before randomisation. Br Med J (Clin Res Ed). 1984;289:1281–1284.
5 Shea S, Bigger JT Jr, Campion J, et al. Enrollment in clinical trials: Institutional factors affecting enrollment in the cardiac arrhythmia suppression trial (CAST). Control Clin Trials. 1992;13:466–486.
6 Cheng SK, Dietrich MS, Dilts DM. A sense of urgency: Evaluating the link between clinical trial development time and the accrual performance of cancer therapy evaluation program (NCI-CTEP) sponsored studies. Clin Cancer Res. 2010;16:5557–5563.
7 Go R, Meyer C, Mathiason M, et al. Nature and outcome of clinical trials conducted by the Eastern Cooperative Group (ECOG) from 1977 to 2006. Poster presented at: American Society of Clinical Oncology; 2010; Chicago, Ill. Abstract 6069.
8 Schroen AT, Petroni GR. Preliminary evaluation of factors associated with premature trial closure and feasibility of accrual benchmarks in phase III oncology trials. Clin Trials. 2010;7:312.
9 Cheng SK, Dietrich MS, Dilts DM. Predicting accrual achievement: Monitoring accrual milestones of NCI-CTEP-sponsored clinical trials. Clin Cancer Res. 2011;17:1947.
11 Durivage H, Bridges K. Clinical trial metrics: Protocol performance and resource utilization from 14 cancer centers. J Clin Oncol. 2009;27(15 suppl):6557.
12 C-Change. Executive summary. In: A Guidance Document for Implementing Effective Cancer Clinical Trials. Version 1.2. Washington, DC: C-Change; 2005.
14 Dilts D. US cancer trials may go the way of the Oldsmobile. Nat Med. 2010;16:632.
15 Dilts DM, Cheng SK, Crites JS, Sandler AB, Doroshow JH. Phase III clinical trial development: A process of chutes and ladders. Clin Cancer Res. 2010;16:5381.
16 Cooper RG, Edgett SJ, Kleinschmidt EJ. Portfolio Management for New Products. 2nd ed. Cambridge, Mass: Perseus Pub; 2001.
17 Staw B. Knee deep in the big muddy. Organ Behav Hum Decis Process. 1976;35:124–140.