Secondary Logo

Journal Logo

Cardiovascular and Thoracic Anesthesiology: Original Clinical Research Report

Achieving Milestones as a Prerequisite for Proceeding With a Clinical Trial

Mickle, Angela M. MS; Maybrier, Hannah R. BS; Winter, Anke C. MD, MSc; McKinnon, Sherry L. BS; Torres, Brian A. MSN, CRNA; Lin, Nan PhD; Lenze, Eric J. MD; Stark, Susan PhD, OTR/L, FAOTA; Muench, Maxwell R. BS; Jacobsohn, Eric MBChB, MPHE, FRCPC; Inouye, Sharon K. MD, MPH; Avidan, Michael S. MBBCh; Wildes, Troy S. MD; on behalf of the ENGAGES Research Group*

Author Information
doi: 10.1213/ANE.0000000000002680
  • Free



  • Question: Can the identification and tracking of key study milestones early in a clinical trial be used to promote the success of a study and demonstrate its feasibility?
  • Findings: Investigators at Washington University in St Louis, MO, were able to select and meet key early study milestones, which suggested long-term feasibility and the potential for significant scientific impact.
  • Meaning: The careful selection and pursuit of key study milestones early in a clinical trial may drive success in study conduct and help to justify continuing the study.

Randomized clinical trials have impacted health care delivery around the world,1 and they provide a foundation for determining the effectiveness of new therapies and innovations.2 However, large clinical trials are expensive. The National Institutes of Health (NIH) invests $30.1 billion in medical research annually, and it is important that such investments translate into scientific discovery and impact health outcomes. Unfortunately, many funded trials do not yield benefits for health care or sufficiently advance science. A recent study of 244 cardiovascular interventional trials funded through the National Heart, Lung and Blood Institute between January 1, 2000, and December 1, 2011, found that fewer than two-thirds of these funded studies published data within 30 months of completion.3 Additionally, many studies are not completed due to infeasibility or futility.4 Many studies have poor study design, are too complex, or are not powered to answer the hypothesis in question. Others yield results that are not translatable to a clinical setting.

As part of continuing efforts to increase the yield of research investments, the NIH recently introduced a phased funding mechanism for clinical trials incorporating 2 distinct periods: the UH2 and UH3.5 During an initial 1-year pilot phase (UH2), investigators are required to reach essential trial milestones. Milestones are selected in collaboration with study stakeholders, including the funding agency, and are designed to address potential barriers that may impede successful study launch, timely trial completion, and the generation of useful scientific results. Attainment of these milestones serves as partial justification for allocating funding for the subsequent, lengthier UH3 phase within which the full trial is performed. Conversely, failing to meet critical milestones during the UH2 phase signifies residual barriers that must be overcome for successful study conduct. Alternative research plans may be necessary and, if barriers cannot be overcome, funding may be best directed toward other potential trials. In total, using this phased mechanism may divert funding to larger numbers of promising pilot studies, while long-term funding is focused on those studies demonstrating a high likelihood of timely completion with the generation of meaningful scientific results.

Investigators at Washington University in St Louis, MO, received funding through the UH2/UH3 mechanism from the National Institute on Aging (NIA) to conduct the Electroencephalography Guidance of Anesthesia to Alleviate Geriatric Syndromes (ENGAGES) study (NCT02241655).6 The ENGAGES study is a pragmatic, randomized controlled trial testing whether electroencephalography (EEG) guidance of general anesthesia can decrease postoperative delirium, a common neurological complication of surgery in older adults, and its downstream negative sequelae. The motivation for this pilot was to demonstrate that the prospectively identified critical study milestones could be met, justifying the pursuit of a larger, pragmatic clinical trial. In this article, we present the methods for determining milestones for the pilot phase of the ENGAGES trial, report successes and failures in reaching milestone goals, and demonstrate how the process provided confidence in and facilitated funding of the expanded 4-year UH3 study phase.


The ENGAGES trial was 1 of 6 studies funded through the UH2/UH3 mechanism. The study was approved by the Washington University institutional review board (IRB), and written informed consent was obtained from all subjects. The NIH appointed an independent research consulting corporation (Westat, Rockville, MD) to evaluate progress during the UH2 period for all 6 individual trials. Investigators from each research team participated in monthly conference calls with Westat and an NIA program official. The agenda of these regular meetings centered on identification of study tasks and barriers, establishing related milestones, and assessment of milestone progress. Milestones served both as a report card for demonstrating candidacy for the next phase and as an unambiguous shared action plan for carrying out a successful pilot study.

Common Thematic Categories for Study Tasks and Barriers, Corresponding Milestones Selected for the ENGAGES Study, and Milestone Outcomes for the ENGAGES Study
Figure 1.
Figure 1.:
CONSORT diagram for the ENGAGES study, indicating the key milestones for the pilot phase. AE indicates adverse event; CAM, Confusion Assessment Method; EEG, electroencephalography; ENGAGES, Electroencephalography Guidance of Anesthesia to Alleviate Geriatric Syndromes; SAE, serious adverse event; SATISFY-SOS, Systematic Assessment and Targeted Improvement of Services Following Year Long Surgical Outcomes Surveys.

Barriers to the launch and conduct of clinical studies often fall within typical themes. The overall trial design, detailed study protocol, and other logistical considerations of the ENGAGES study were examined for such barriers. Those most likely to delay launch, prevent timely completion, or compromise the scientific validity of the ENGAGES study were incorporated into milestones (Figure 1). Common thematic categories for these barriers across studies and the corresponding ENGAGES milestones are described as follows (Table):

  1. Standardization, transparency, and clearances: Sound research practice requires adherence to investigative conventions.7 Several considerations must be addressed to minimize bias, control undesired variation, promote transparency,8 and facilitate reproducibility.9
  2. A study protocol and manual of procedures should be prepared, including case report forms, data collection source documents and processes, standardized definitions, and delegations of trial conduct responsibility. Other regulatory documents may be required depending on trial features (eg, Food and Drug Administration form 1572 for new investigational drug studies). Trial end points and data analysis plans should be prespecified.10
    The trial should be registered in the scientific community (eg, for example at with inclusion of the aforementioned elements to promote transparency of methods, protocol adherence, sound analysis practices, and shared scientific knowledge.11 To promote the highest level of transparency and knowledge sharing, the study protocol should be published or otherwise disseminated early in the launch process, even for observational studies.12
    Specific clearances are needed before study launch to promote patient welfare. These include approval from an institutional review board, institutional clearances, and, depending on the nature of the study, additional authorities may need to provide approval, such as an investigational pharmacy or a radiology committee.
    The prompt shared recognition of necessary prestudy documents and clearances helps to facilitate research team communication and avoid delays. In particular, the ENGAGES study team identified preparation of the detailed study protocol, finalization of data collection tools, public registration of the study, and obtaining IRB approval as key barriers within this domain.
  3. Recruitment: The ability to enroll a sufficient number of eligible patients over a projected time interval is a common barrier to study conduct. The recruitment start date, recruitment completion date, and requisite average rate of subject recruitment should be plainly identified. Studies often launch with an attenuated recruitment rate to optimize successful study conduct, but the early phases of a study must also be leveraged to determine whether the recruitment rate necessary for the full trial is realistically achievable.13 Additionally, because clinical trial drop-out rates can be as high as 30%,14 attrition of patients between recruitment and the completion of outcome assessments should be evaluated. If recruitment is inadequate or outcome collection is compromised, contributing etiologies must be identified and addressed.
  4. For the postpilot phase of the ENGAGES study, a sustained randomization rate of at least 32 patients per month would be needed to complete the trial within the funding period. To permit early familiarization of the research team and clinical staff to study processes, an approximate 50% recruitment rate of 12–16 patients randomized per month (66 total patients during the 6-month pilot study period) was selected as the recruitment milestone for the pilot period. For the later pilot period, it was collectively recognized that surpassing this milestone and approaching the necessary long-term rate of 32 patients per month would further allay any concerns regarding the long-term feasibility of trial recruitment.
  5. Study intervention: In clinical research, and particularly in large and pragmatic clinical trials, the application of study treatments may depend on factors such as clinical personnel with diverse beliefs, expertise, and usual practices. Such diversity is likely to introduce variation that must be recognized, and if appropriate, mitigated. Trials designed with an explanatory approach generally aim to deliver intervention and control treatments with high fidelity so that the effect of the intervention can be accurately assessed.15 In contrast, pragmatic studies attempt to study the application of treatments in real world settings that are accompanied by greater variation in delivery.15 Still, overlooking practical standardization methods in pragmatic studies, such as providing basic education to clinicians about treatment protocols, may compromise trial results. Beyond the importance of ensuring high-fidelity treatment application for generating useful results within the study, detailed descriptions of any standardized methods for treatment delivery are critical for replication in subsequent studies or application to clinical medicine.
  6. Any study presents unique barriers to high-fidelity delivery of trial interventions. Considerable foresight is required to anticipate and overcome these barriers. Advance consideration must be given to sources of treatment variation. Measures to manage undesired treatment variation may be introduced,16 depending on the study design and trial objectives.17 Action plans may include training of clinicians applying treatments, provision of reference materials, checklists, adherence reminders, quality assurance monitoring, and summative adherence reporting. In addition to the introduction of prospective measures, intervention fidelity should be monitored throughout the trial so that undesired sources of variance can be promptly identified and mitigated as they appear.16 Such measures should be considered and applied in accordance with the general design of the trial. For example, for a largely explanatory trial, more aggressive measures would be implemented, whereas for a pragmatic trial, less behavioral modification would be pursued.
    For the ENGAGES study, anesthesiology clinicians are tasked with dosing anesthesia using a novel EEG-guided protocol versus routine anesthetic care. Using input from study stakeholders, a plan for expeditious and reproducible training of diverse anesthesiology clinicians on intraoperative EEG interpretation was devised. Practical methods for promoting knowledge retention were also selected. Specific methods were prioritized because they would be feasible to replicate within future studies or clinical care. Many anesthesiology clinicians were trained in EEG waveform interpretation and the EEG-guided anesthesia protocol through a focused lecture. The training session used a skills test to assess efficacy. These components were then embedded in a computer-based module for ongoing use. Additional web-based instruction ( and paper-based teaching materials were prepared to promote maintenance of knowledge. Quick reference cards were prepared that could be easily used during the course of clinical care. Additionally, before each ENGAGES operative case, an email was sent to the anesthesiology clinicians with a short reminder including the principals of treatment group assignment, a brief reference guide, and links to refresher materials. Research team members were made available in the perioperative period to answer additional questions regarding the EEG-guided anesthetic protocol. For each research patient, anesthesiology clinicians were asked to complete an adherence checklist assessing whether they had adhered to treatment group principals, whether any barriers had impeded carrying out the assigned treatment, and whether any complications had occurred during the surgical procedure that could be related to the assigned treatment. Additionally, to assess the treatment group fidelity, differences in volatile anesthetic concentrations and EEG suppression were assessed between the treatment and control groups in the pilot cohort (ie, age-adjusted fraction of minimum alveolar anesthetic concentration and elapsed time with a suppressed EEG waveform). These educational tasks and assessment measures were encapsulated in pilot phase milestones.
  7. Outcome and data collection: Accurate, reliable, and maximally complete collection of outcome data is necessary for a scientifically useful trial. Because outcome data may be subject to laboratory instrument, research personnel, or patient report variation, proactive efforts to promote precise data collection are important. It is additionally important to institute up-front strategies for minimizing missing outcome data and the loss of patients to follow-up. The mechanisms for complete and accurate data collection should be clearly defined and demonstrated before study launch. Customized case report forms and electronic databases must be created to standardize data collection.18 Thresholds for the acceptability of missing study data should be determined, recognizing the feasibility of collecting specific study measures. Establishment of transparent milestones and a monitoring plan for missing data allow the research team to recognize gaps in data collection, enabling early adjustments in data collection procedures.
  8. In the ENGAGES study, the primary outcome of delirium is determined using the Confusion Assessment Method (CAM) tool, an assessment modality that requires training. To ensure that study staff were able to accurately obtain the primary outcome measure, a comprehensive training program on the CAM tool was defined and instituted for research personnel. Thereafter, investigators measured both interrater agreement and test–retest reliability for CAM assessments performed by the research team. Additionally, because delirium has acute onset and often fluctuates, it was deemed important to assess patients for delirium daily for 5 postoperative days to maximize the detection of incident postoperative delirium. Finally, the ability to reliably collect 30-day secondary outcomes data from discharged postoperative patients was identified as a potential barrier. These aspects of trial conduct were assessed in the UH2 milestones.
  9. Patient safety: Patient safety is the primary consideration when conducting any clinical trial. The establishment of a data safety monitoring board (DSMB) before study launch is essential. A DSMB charter must be prepared and approved, defining the responsibilities of the DSMB and procedures for reviewing safety data. The investigator and DSMB must together ensure that the required safety data are being collected and reviewed in a timely fashion through construction of a prespecified DSMB plan.19 Thoughtful communication among the DSMB members and investigators before study launch is essential to ascertain potential risks, ensure that safety data are reliably collected, construct plans for review of safety data, and establish a protocol for responding to safety data. These considerations were incorporated in the UH2 milestones.
  10. A comprehensive list of critical milestones relating to the above categorization of study tasks and barriers were collaboratively selected by the investigative team, the funding stakeholder (NIA), and the independent consulting organization (Westat; Table).


At the end of the UH2 phase of the ENGAGES trial, investigators were successful in completing the predefined study-specific milestones, and were subsequently awarded additional funding for the UH3 phase. The outcomes of milestones selected for the pilot phase of the UH2 phase of the ENGAGES trial were as follows (Table).

Standardization, Transparency, and Clearances

The study protocol, detailed study manual of operations, and data collection forms were completed by the target dates. IRB approval of the study (August 2014) and the necessary regulatory documents were completed before recruitment in the trial, which began in January 2015. To facilitate transparency, the trial was registered on in December 2014, and the study protocol was submitted for publication in a peer-reviewed journal.6


Figure 2.
Figure 2.:
Randomization rates during the UH2 phase and projections for the UH3 phase. CONSORT indicates Consolidated Standards of Reporting Trials; ENGAGES, Electroencephalography Guidance of Anesthesia to Alleviate Geriatric Syndromes.

Over the 25-week UH2 pilot period, 117 patients were randomized, producing an overall recruitment rate of 19.5 patients per month. An additional 30 patients were enrolled but did not proceed to surgery. This exceeded the randomization goal of 66 patients (10–14 patients per month). Figure 2 illustrates the randomization milestone for the pilot study, the randomization rate that would be needed for the expanded (postpilot) study phase, and the actual randomization rate that was observed during the UH2 phase. In the later pilot period, a comparative analysis of the ongoing recruitment rate and future recruitment goal of 32 patients per month was performed. Research team resources and additional subjects were identified that would enable the higher recruitment rate to be achieved during the UH3 phase.

Study Intervention

Several initiatives were performed to promote and assess the fidelity of the study treatments, including developing training modules, deploying checklists, and assessing physiological data from the pilot study subjects. Aggregate data from the 117 patients randomized in the UH2 phase showed: (1) a reduction in volatile anesthetic concentration in the EEG-guided treatment group relative to the control group (median age–adjusted minimum alveolar anesthetic concentration, 0.78 vs 0.93, respectively; P < .001); (2) a reduction in EEG suppression in the EEG-guided group (median proportion of time with zero EEG suppression, 94.0% vs 87.0%; P = .008); (3) and higher median bispectral index values in the EEG-guided group (45.5 vs 42.0; P = .002). Thus, the aggregate pilot data supported that the EEG-guided protocol resulted in anesthetic administration alterations, and these were associated with attendant decreases in EEG suppression.

Outcomes and Data Collection

The 38 concurrent interrater delirium assessments demonstrated almost perfect agreement (κ = 0.94 [95% CI, 0.86–1]). The strong interrater agreement indicates excellent psychometric properties of the CAM tool as used by the research team after the training program. Test–retest reliability of 35 assessments was performed by 2 separate assessors interviewing the same patient approximately 2 hours apart. This revealed a κ statistic of 0.53 (95% CI, 0.24–0.83), indicating moderate agreement. The moderate test–retest agreement obtained over time in individual patients is unsurprising, given the fluctuating nature of delirium.

After pilot enrollment commenced, key aggregate outcomes were assessed to ensure that outcomes data would be complete and enable valid results. Within the 117 patients randomized during the pilot period, CAM assessments were successfully performed on 98.3% of patients and revealed a delirium incidence of 28.1%. This is consistent with rates estimated from the literature,20 which were used in the trial sample size power calculations. Ninety-eight patients were eligible to be contacted for 30-day follow-up during the UH2 phase, and multiple methods were used to ensure high follow-up rates, including phone calls, emails, letters, and contacting family members. A total of 84 patients (85.7%) completed at least 1 survey approximately 30 days after surgery; 3 declined; 2 patients died; and 9 patients did not respond. For the 9 nonresponders, the 30-day vital status (whether alive or dead) was obtained for 6 patients. This exceeded our prespecified goal of >85% follow-up rate at 30 days.

Patient Safety

The DSMB was established on time for study launch, and the DSMB charter and safety assessment plans were approved. As outlined in the safety monitoring plan, all adverse events that occurred within 30 days of the study intervention were initially reviewed by study investigators to determine whether they were possibly related to the trial intervention. Based on the recommendation of the DSMB, the ENGAGES team also hired an unbiased safety officer to review any serious adverse events to determine relatedness to the study. Based on aggregated data from the pilot study, the safety officer, DSMB board, and ENGAGES investigators all concluded that there was no evidence of study-related harm to ENGAGES participants.


This article provides general guidelines for rigorous conduct of large clinical trials, and specifically highlights the utility of adopting a phased approach incorporating trial-specific milestones. The milestones important to the long-term success of the ENGAGES trial were identified, tracked, and achieved during a pilot funding year (UH2) using the novel phased funding mechanism (UH2/UH3) introduced by the NIH. The execution of important UH2 phase milestones demonstrated the feasibility of the study during the pilot year and provided motivation for the ongoing funding of the multiyear study phase (UH3) of the trial. Challenges typically encountered with study implementation were assessed for those that would be most informative to the success of the ENGAGES study. By ensuring that the milestone selection process was collaborative, all stakeholders had a voice in ensuring that the aspects of the study they felt were most at risk for impeding success would be carefully tracked. Therefore, when all the selected milestones were successfully achieved during the initial year of funding, stakeholders were reassured that the study was feasible and that it was expected to generate useful scientific findings in a timely fashion. The ENGAGES study is currently in the midst of its 4-year expanded UH3 funding phase, with an expected trial completion date of June 2019. Several of the UH2 phase milestone measures, such as recruitment rate tracking and treatment fidelity monitoring, have been implemented into continued monitoring during the long-term UH3 study phase.

Widespread application of milestone processes into large, multiyear trials may promote higher rates of trial completion, earlier dissemination of knowledge, and fewer trials that do not produce scientifically useful results. Failures in these areas may occur for several reasons. For example, in large pragmatic trials such as the ENGAGES study critical problems may include: (1) statistical planning that does not reflect actual primary outcome incidences in the study population; (2) inadequate research infrastructure to reliably collect study data; (3) insufficient ongoing enrollment to meet study objectives in a timely fashion; (4) failure to implement methods for ensuring reliable administration of study treatments; and (5) failure to complete regulatory or define study processes on time for an expeditious study launch. Collaborative consideration by study stakeholders such as patient representatives, clinicians, and researchers may be able to anticipate the majority of these potential barriers in advance. Early recognition allows for shared problem-solving and may enable identification of flaws in study design. When major obstacles or limitations are identified, the study should either be adapted or abandoned to avoid further wasted investments by patients, researchers, clinicians, and funding agencies. Conversely, there is a potential concern that the milestone process could increase waste because studies that are likely to succeed might be unnecessarily terminated at the end of the pilot phase. Therefore, it is essential even when all milestones are not accomplished, to assess the risks and benefits of continuing the study, and to consider whether barriers might be surmountable.

Despite the apparent successful application of the UH2/UH3 milestone-based phased funding approach in the ENGAGES trial described here, the approach has several potential limitations that must be clarified by further study. First, the utility of the milestone process is likely heavily dependent on accurate identification of critical milestones early in the study process. Although some metrics (eg, adequate rates of subject enrollment) are applicable across all studies, individual studies face unique challenges in most other aspects of their design and launch. Failure to successfully identify even a single critical challenge early in the study may compromise the utility of the entire milestone approach. We believe the involvement of a broad coalition of study stakeholders may help to circumvent such a problem.

The role of the milestone process in identifying or mediating successful trials requires additional study. In the ENGAGES trial, the use of the UH2 milestone process was associated with expeditious study launch, successful continued funding, the overcoming of key study barriers, and continued successful study conduct in the early UH3 study phase. However, to what extent the milestone process mediated these early successes is not known. Additionally, the impact of the milestone process on late study metrics must be clarified. These include timely study completion, generation of useful scientific data, rapid dissemination of results, and applicability of study interventions to real world care. It remains possible that the short-term metrics reported in this current study may not be sufficiently associated with success in the more important long-term metrics. Additionally, it remains possible that previously unrecognized pitfalls might arise later during the study period and compromise study conduct.

Importantly, a companion trial has been launched in Canada (ENGAGES-Canada, NCT02692300). ENGAGES-Canada investigators have similarly created their own study-specific milestones to further evaluate success at the pilot phase and to inform whether or not to proceed with a larger trial. Although many milestones may be similar to those described in this article, it is envisaged that Canadian conditions and stakeholders will identify some different challenges to the conduct of the trial.

In summary, during the pilot of the ENGAGES study, the UH2/UH3 milestone-based phased funding process was associated with successful navigation of early trial barriers and competition for continued funding. Additionally, all study barriers that became apparent by the completion of the pilot period were anticipated through early collaborative discussions among study stakeholders. The long-term utility of study milestone processes requires further research. If the utility of such processes is demonstrated across trials, such an approach could become an invaluable tool for reducing wasted resources and promoting high yield from trial investments on the part of patients, funding institutions, clinicians, and researchers.


The authors thank Molly Wagster (National Institute on Aging), members of the data safety monitoring board (Frederick Sieber, Heather Allore, Donna Marie Fick, Charles W. Hogue, Pratik Pandharipande, and Charles Brown), Westat representatives (Nora Dluzak, Paula Darby Lipman, and Mary Masters).

Members of the ENGAGES Research Group are as follows: G. P. Apakama, M. S. Avidan, D. A. Emmert, K. E. Escallier, B. A. Fritz, T. J. Graetz, R. Hueneke, S. K. Inouye, E. Jacobsohn, E. J. Lenze, J. M. Leung, N. Lin, H. R. Maybrier, S. L. McKinnon, S. J. Melby, A. M. Mickle, M. R. Muench, M. R. Murphy, B. J. Palanca, E. M. Schmitt, S. Stark, T. W. Stevens, B. A. Torres, R. T. Upadhyayula, T. S. Wildes, and A. C. Winter; members of the ENGAGES-Canada Research Group: M. Monterola, M. Kavosh, R. Legaspi, R. El-Gabalway, K. Aquino, L. Girling, and E. Jacobsohn.


Name: Angela M. Mickle, MS.

Contribution: This author helped with writing the manuscript, data acquisition, and analysis.

Name: Hannah R. Maybrier, BS.

Contribution: This author helped with editing the manuscript and data acquisition.

Name: Anke C. Winter, MD, MSc.

Contribution: This author helped with conceptualizing study design and editing the manuscript.

Name: Sherry L. McKinnon, BS.

Contribution: This author helped with conceptualizing study design and editing the manuscript.

Name: Brian A. Torres, MSN, CRNA.

Contribution: This author helped with conceptualizing study design and editing the manuscript.

Name: Nan Lin, PhD.

Contribution: This author helped with conceptualizing study design, data analysis, and editing the manuscript.

Name: Eric J. Lenze, MD.

Contribution: This author helped with conceptualizing study design and editing the manuscript.

Name: Susan Stark, PhD, OTR/L, FAOTA.

Contribution: This author helped with conceptualizing study design and editing the manuscript.

Name: Maxwell R. Muench, BS.

Contribution: This author helped with data acquisition and editing the manuscript.

Name: Eric Jacobsohn, MBChB, MPHE, FRCPC.

Contribution: This author helped with conceptualizing study design and editing the manuscript.

Name: Sharon K. Inouye, MD, MPH.

Contribution: This author helped with conceptualizing study design and editing the manuscript.

Name: Michael S. Avidan, MBBCh.

Contribution: This author helped with conceptualizing study design and editing the manuscript.

Name: Troy S. Wildes, MD.

Contribution: This author helped with conceptualizing study design and writing the manuscript.

This manuscript was handled by: W. Scott Beattie, PhD, MD, FRCPC.


1. Lauer MS, D’Agostino RB Sr. The randomized registry trial–the next disruptive technology in clinical research? N Engl J Med. 2013;369:1579–1581.
2. Antman EM, Harrington RA. Transforming clinical trials in cardiovascular disease: mission critical for health and economic well-being. JAMA. 2012;308:1743–1744.
3. Gordon D, Taddei-Peters W, Mascette A, Antman M, Kaufmann PG, Lauer MS. Publication of trials funded by the National Heart, Lung, and Blood Institute. N Engl J Med. 2013;369:1926–1934.
4. Ioannidis JP. Why most clinical research is not useful. PLoS Med. 2016;13:e1002049.
5. Lauer MS, Bonds D. Eliminating the “expensive” adjective for clinical trials. Am Heart J. 2014;167:419–420.
6. Wildes TS, Winter AC, Maybrier HR, et al. Protocol for the Electroencephalography Guidance of Anesthesia to Alleviate Geriatric Syndromes (ENGAGES) study: a pragmatic, randomised clinical trial. BMJ Open. 2016;6:e011505.
7. Clinical Trials Guidance Documents. US Food and Drug Administration. Available at: Published 2016. Accessed July 29, 2016.
8. Ioannidis JP. How to make more published research true. PLoS Med. 2014;11:e1001747.
9. Collins FS, Tabak LA. Policy: NIH plans to enhance reproducibility. Nature. 2014;505:612–613.
10. Thomas L, Peterson ED. The value of statistical analysis plans in observational research: defining high-quality research from the start. JAMA. 2012;308:773–774.
11. Dal-Ré R, Ioannidis JP, Bracken MB, et al. Making prospective registration of observational research a reality. Sci Transl Med. 2014;6:224cm1.
12. Eisenach JC, Kheterpal S, Houle TT. Reporting of observational research in anesthesiology: the importance of the analysis plan. Anesthesiology. 2016;124:998–1000.
13. Alexander W. The uphill path to successful clinical trials: keeping patients enrolled. P T. 2013;38:225–227.
14. National Academy of Sciences. The Prevention and Treatment of Missing Data in Clinical Trials. 2010. Washington, D.C.: National Academies Press. Available at: Accessed March 1, 2016.
15. Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Clin Epidemiol. 2009;62:499–505.
16. Horner S, Rew L, Torres R. Enhancing intervention fidelity: a means of strengthening study impact. J Spec Pediatr Nurs. 2006;11:80–89.
17. Treweek S, Zwarenstein M. Making trials matter: pragmatic and explanatory trials and the problem of applicability. Trials. 2009;10:37.
18. Richesson RL, Nadkarni P. Data standards for clinical research data collection forms: current status and challenges. J Am Med Inform Assoc. 2011;18:341–346.
19. Woodin KE. Patient safety comes first in clinical trials. Medscape Viewpoint. Available at: Published 2000. Accessed September 2016.
20. Vasilevskis EE, Han JH, Hughes CG, Ely EW. Epidemiology and risk factors for delirium across hospital settings. Best Pract Res Clin Anaesthesiol. 2012;26:277–287.
Copyright © 2017 International Anesthesia Research Society