Secondary Logo

Journal Logo

E-SCOPE

A Strategic Approach to Identify and Accelerate Implementation of Evidence-based Best Practices

Henry, Shayna L. PhD*; Mohan, Yasmina MPH*; Whittaker, Joel L. MPH*; Koster, Marguerite A. MA, MFT*; Schottinger, Joanne E. MD*; Kanter, Michael H. MD

doi: 10.1097/MLR.0000000000001191
Original Articles
Open

Background: More than 50,000 randomized controlled trials and 8000 systematic reviews are anticipated to be published annually in the coming years. This huge volume of published findings makes it challenging for health care delivery systems to review new evidence, prioritize health care practices that warrant implementation, and implement best practices.

Objective: The objective of this study was to describe the Kaiser Permanente Southern California E-SCOPE (Evidence Scanning for Clinical, Operational, and Practice Efficiencies) program, a systematic method to accelerate the implementation of evidence-based practices in clinical care settings.

Methods: E-SCOPE uses a strategic evidence search algorithm to conduct proactive literature searches to identify high-quality studies of interventions that yield improved health outcomes, quality and/or efficiency of care delivery, or cost savings. Each quarterly search yields 500–1000 abstracts; about 5%–10% of studies are selected each quarter for consideration for implementation. These studies are presented to clinical and operational leaders and other stakeholders to make the final determination regarding the implementation of the practice; E-SCOPE staff work closely with stakeholders to develop an implementation plan, identify practice owners, and ensure sustainability.

Results: The time from study publication to implementation using the E-SCOPE process ranges from 4 to 36 months, with an average of ∼16 months. Four examples of E-SCOPE implementation efforts, including new deployment, scale-up/spread, deimplementation, and operational efforts, are described.

Conclusion: A single, centralized program for the proactive identification of the most up-to-date, evidence-based best practices and facilitated implementation can efficiently and effectively promote continuous learning and implementation in a learning health care system.

*Evidence-Based Medicine Services Unit, Department of Clinical Analysis, Kaiser Permanente Southern California

Southern California Permanente Medical Group, Department of Quality & Clinical Analysis, Pasadena, CA

Supported only by internal operating budgets in the Southern California Permanente Medical Group Department of Clinical Analysis.

Portions of this manuscript have been presented at the Kaiser Permanente Center for Effectiveness and Safety Research Annual Meeting 2017, the Kaiser Permanente National Quality Conference 2018, the Southern California Clinical and Translational Science Institute Regional Dissemination Implementation & Improvement (DII) Science Symposium 2018, the Health Technology Assessment International Annual Meeting 2018, and the Institute for Healthcare Improvement Annual Meeting 2018.

The authors declare no conflicts of interest.

Reprints: Shayna L. Henry, PhD, Department of Clinical Analysis, Kaiser Permanente Southern California, 393 East Walnut Street, Pasadena, CA 91188. E-mail: shayna.l.henry@kp.org.

This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal. http://creativecommons.org/licenses/by-nc-nd/4.0/

Nearly 2.5 million publications were added to PubMed in 20171; nearly 60,000 new randomized controlled trials (RCTs) and systematic reviews are anticipated to be published annually in the coming years (http://blogs.trusttheevidence.net/). This growing volume of publications makes it challenging for health care providers and delivery systems to critically appraise newly published evidence and translate this evidence into practice.

Health care quality improvement professionals are familiar with the adage that it takes, on average, 17 years for published evidence to be disseminated and implemented in clinical practice.2 This time lag between publication and implementation is a considerable challenge to providing timely high-quality, evidence-based health care.3 Historically, implementation has relied primarily on provider education and the production of clinical guidelines.3 Although these are critical elements of implementation, they are not the only approaches necessary to efficiently assess the evidence and embed it into clinical practice.

The path from evidence generation to effective implementation is not linear—the pipeline is fractured and requires dedicated resources to address issues related specifically to the implementation of new or updated practices.4,5 The implementation of new, evidence-based practices—or the “deimplementation” of practices for which evidence is lacking—requires that stakeholders know about the evidence, understand it, and appropriately interpret the findings for use.6 A broader, more proactive and systematic approach is needed to continually move up-to-date evidence into the health care delivery setting.3 The Kaiser Permanente Southern California Evidence Scanning for Clinical, Operational, and Practice Efficiencies (E-SCOPE) program represents a systematic and effective way to break the logjam that slows the real-world implementation of evidence-based practices in the clinical care setting by proactively identifying and conducting critical appraisals of the evidence, confirming that it meets both scientific standards and system-specific needs, and working closely with clinical and operational stakeholders to determine optimal implementation strategies.

The E-SCOPE program addresses the challenges of identification and timely implementation of evidence-based best practices by expediting the deployment of practices that: (1) have a rigorous evidence base; (2) have not yet been implemented or are underutilized at Kaiser Permanente Southern California; (3) improve quality, safety, timeliness, and/or efficiency of patient care; and (4) are likely to be both sustainable and cost-effective. Below, we describe the E-SCOPE process and several examples of ongoing or recently completed E-SCOPE projects to illustrate effective operation in a large learning health care organization.

Back to Top | Article Outline

METHODS

Kaiser Permanente Southern California is a fully integrated health maintenance organization that provides comprehensive health services to >4.5 million health plan members at 15 medical centers across the region. In 2014, Kaiser Permanente Southern California quality leaders requested that their Evidence-Based Medicine (EBM) Services Unit develop a system for monitoring and disseminating high-quality published studies of effective health care interventions to keep clinical leaders apprised of recently published studies and identify effective health care practices that warrant implementation. This request originated with the realization that the system’s existing performance-improvement activities yielded too few new, effective practices warranting systemwide implementation, even fewer of which were evidence based. E-SCOPE was initiated by tasking a Senior EBM Specialist in critically appraising scientific evidence, with developing an algorithm-based search strategy to identify new high-quality clinical trials and systematic reviews of promising practices that were not in wide use at Kaiser Permanente Southern California. The studies and their findings were forwarded to regional chiefs of relevant specialty departments. However, because chiefs and other stakeholders had limited time and resources to implement new practices, this single strategy proved insufficient to move identified interventions into practice, and an implementation project manager was added to the E-SCOPE team to assist with implementation efforts. This compound strategy of support for practice identification and implementation proved effective, increasing the number of E-SCOPE initiated projects, and a second implementation project manager was added to better manage the program’s portfolio.

The E-SCOPE process (Fig. 1) was previously described by Kanter et al.7

FIGURE 1

FIGURE 1

Back to Top | Article Outline

Phase 1: Conduct Strategic, Algorithm-based Quarterly Evidence Searches and Screen/Select Studies

The E-SCOPE team is comprised of 3 dedicated staff members (1 Senior EBM Specialist and 2 Implementation Project Managers) and 4 Kaiser Permanente Southern California Regional Quality Leaders (the Medical Director and Assistant Medical Directors of Quality and the EBM Services Senior Manager). The EBM Specialist and 1 project manager use an internally-developed strategic search algorithm (Fig. 2) to conduct quarterly searches of the biomedical literature to identify studies of interventions that report moderate to high impact on important health outcomes, cost savings, and/or improvements in care processes and efficiency. When reviewing abstracts, priority is given to high-quality systematic evidence reviews and RCTs. Additional priorities include studies reporting the significant absence of impact that suggests discontinuing care practices without benefit. In addition, the generalizability of studies to Kaiser Permanente Southern California is a critical factor.

FIGURE 2

FIGURE 2

Each quarterly search generates between 500 and 1000 abstracts; ∼10%–20% of abstracts meet initial, predetermined criteria for study quality and feasibility within Kaiser Permanente Southern California. These study quality criteria are illustrated in Figure 2, and are based on EBM tools such as the Cochrane “Risk of Bias” tool,8 aspects of the Grading of Recommendations Assessment, Development and Evaluation framework9 (GRADE), and A MeaSurement Tool to Assess systematic Reviews10 (AMSTAR). E-SCOPE’s regional quality leaders then review the screened abstracts, considering whether interventions are already in practice, alignment with patient values and expectations, the balance of benefits and potential harms, and resource implications. On average, 60%–70% of screened abstracts are moved forward following this review. The full texts of the remaining studies are then more thoroughly reviewed by E-SCOPE staff; an additional 10%–25% of studies are typically eliminated from consideration due to concerns regarding their rigor or implementation feasibility. In any given quarter, around 5%–10% of the initial search results meet E-SCOPE standards for dissemination to stakeholders and consideration for potential implementation. Finally, while the criteria illustrated in Figure 2 are the basic standards, E-SCOPE staff do exercise some discretion in their selections, reflective of institutional preferences and norms.

Back to Top | Article Outline

Phase 2: Decide Which Evidence-based Practices to Implement

Stakeholder support is crucial for adopting study interventions. Although E-SCOPE’s quality leaders may recommend implementation or deimplementation, clinical and operational leaders and other stakeholders at both the regional and local levels determine what moves forward. These stakeholder groups, including chiefs of service in relevant specialty department(s), committee chairs, and quality improvement staff determine whether the identified practices are suitable for integration into existing clinical or operational pathways or require new workflows. Where there are disagreements between the quality leaders and stakeholders, a robust discussion ensues, and the clinical/operational leaders are expected to be able to articulate why a practice cannot or should not be implemented. Practices that are given the “green light” by stakeholders are then put on a path to implementation, and the E-SCOPE implementation project managers work to assemble and guide a multidisciplinary team of clinical and operational stakeholders that assumes responsibility for implementing the evidence-based practice, using processes tailored to the intervention, setting, patient population, and available resources (see Fig. 1, steps 5 and 6).

Back to Top | Article Outline

Phase 3: Support Implementation of Selected Practices

Successful implementation of an initiative across all Kaiser Permanente Southern California medical centers hinges largely on whether it can optimize existing processes or whether it requires new systems, supplies, workflows, or significant changes in provider practice and behavior. E-SCOPE implementation project managers work directly with practice stakeholder teams to develop an implementation plan for each project. The structure of the project plan depends on the complexity of the intervention, but all include the following elements: evidence summary, intervention overview, implementation resources, and implementation measurement. For each project, E-SCOPE implementation project managers rely on a suite of tools to facilitate implementation or deimplementation of practices.11 These include leveraging the support of quality leaders, conducting needs assessments of local sites, developing communications for frontline providers (eg, memoranda, clinical protocols, job aids), conducting educational sessions for frontline providers, providing supportive supervision (ie, periodic, agreed-upon check-ins), and utilizing data drawn from our electronic health record (EHR) to conduct audit and feedback cycles that allow for identification of practice gaps and motivate behavioral change. These facilitative tools are described in greater detail in Figure 3.

FIGURE 3

FIGURE 3

Although the E-SCOPE implementation project manager provides oversight, assesses facilitators and barriers to implementation, and helps to troubleshoot challenges, stakeholders are encouraged to take ownership of implementation projects from the outset and spearhead the integration of the new practices into clinical or operational workflows. Although the time from publication to implementation using the E-SCOPE approach varies depending on the complexity of the intervention and multidisciplinary nature of the stakeholder groups, the implementation of new practices ranges from 4 months to 36 months, with an average around 16 months (see Fig. 1, step 7).

Back to Top | Article Outline

Phase 4: Monitor Progress

Importantly, since it is not our charge to replicate the findings of published trials, monitoring of E-SCOPE projects does not call for additional research or formal data analysis, nor does it constitute true program evaluation. Instead, monitoring of E-SCOPE efforts is focused on the degree of implementation and/or spread of the practice rather than on the health outcomes associated with the practice. With only a few exceptions, projects are monitored only for a finite period of time, stopping when stakeholders report, or the data reflect, that practice is stable and has become part of the clinical workflow. Any ongoing monitoring efforts must be borne by the clinical groups involved in carrying out the intervention. In some cases, to accomplish this, E-SCOPE projects have been adopted as regional quality improvement goals in a given specialty, enabling more resources to be directed at tracking implementation and outcomes jointly. Sustainability is achieved in part through monitoring the status of implementation of a given practice—for example, how many babies are getting probiotics? how many imaging studies are still being ordered? If monitoring data indicates a lack of progress, E-SCOPE implementation project managers—with support from Quality Leaders—revisit the practice implementation plan and address shortcomings with local stakeholders (Fig. 311). Such monitoring only takes place until the intervention becomes a stable part of the practice.

Back to Top | Article Outline

RESULTS

To date, E-SCOPE has accelerated the implementation of 30 evidence-based practices (Table 1). Projects represent a few distinct types of implementation—deployment of new practices not currently in use in Kaiser Permanente Southern California, scale-up/spread of underutilized practices, deimplementation of practices for which there is no evidence for continued use, and occasionally savings-focused operational interventions. Below, we briefly describe a representative project from each of these categories.

TABLE 1

TABLE 1

Back to Top | Article Outline

Selected Case Studies: E-SCOPE Implementation Projects

New Deployment: Weight Management For Reducing the Severity of Psoriasis

Evidence indicates that weight management interventions that achieve 3%–14% weight loss among individuals with psoriasis and a body mass index ≥30 can reduce psoriasis severity up to 75%.12 E-SCOPE team members shared this evidence with primary stakeholders—Chiefs of Dermatology—who expressed interest in developing an intervention to promote weight loss among individuals with psoriasis and body mass index ≥30, and to improve patient quality of life by reducing their psoriasis severity and potential consequent need for psoriasis pharmacologic therapy, which could also serve to reduce health care costs. Dermatology Chiefs set their 2018 clinical goal to be the referral of at least 50% of all eligible health plan members to Kaiser Permanente Southern California, weight management group classes. Dermatologists were educated regarding the evidence and trained in motivational interviewing to equip them with the knowledge and skills needed to motivate members to initiate a weight loss action plan. Physicians were supported by our health education department, which provided additional weight management resources such as exam room and waiting room posters, a patient video testimonial, and handouts tailored to individuals with psoriasis, but most importantly the weight management group classes. Referral to weight management classes was streamlined in the EHR system, facilitating the enrollment of patients by their dermatologists at the point of care. In addition, our outreach department identified patients who met criteria and issued them letters or emails (depending on patient preference) to raise awareness about the impact weight loss can have on reducing the severity of psoriasis, and deliver information about how to enroll in Kaiser Permanente Southern California, weight management classes. The same information was also made available to applicable patients via the Kaiser Permanente Southern California, online patient portal.13

To help providers and health educators stay informed, a dashboard was developed to track the number of patients referred for weight management, pounds lost, and prescriptions ordered for pharmacologic treatment for psoriasis. In the first year of the program, patients with psoriasis who attended Kaiser Permanente Southern California, weight management classes had a mean weight loss of 3.72% and psoriasis drug costs dropped an average of $215 per individual.

Back to Top | Article Outline

Scale-Up: Probiotics to Prevent Necrotizing Enterocolitis in Preterm Infants

In 2015, the E-SCOPE team identified numerous systematic reviews of trials14–19 which reported that probiotics for preterm infants in the newborn intensive care unit (NICU) results in lower rates of necrotizing enterocolitis [relative risk (RR)=0.43; number needed to treat (NNT)=30; P<0.00001], lower mortality (RR=0.65; NNT=40; P<0.00001), and lower rates of sepsis (RR=0.86; P=0.007; NNT=44). E-SCOPE team members worked with NICU physician directors to develop an implementation plan, which began with the endorsement of an initial roll out at 5 medical centers. A Regional Probiotic Procedure Protocol was developed and approved by the NICU Nurse Peer Group and NICU Physician Directors and was also shared with physician partners in Kaiser Permanente’s Northern California Region.

In 2017, probiotic administration was added to the NICU admissions order set in the integrated electronic medical record and can be ordered by NICU providers at any Kaiser Permanente Southern California medical center NICU. Prescribing providers order probiotics from their local Kaiser Permanente Southern California inpatient pharmacy, which dispenses the treatment in a single-dose syringe to be given during feedings while in the NICU. Probiotic administration has increased from 2.4% of eligible infants in 2016 to 36.8% of eligible infants in 2017. Monitoring of this project is ongoing as the practice continues to spread.

Back to Top | Article Outline

Deimplementation: Elimination of Continuous Passive Motion Following Total Knee Arthroplasty

A Cochrane systematic review20 of 24 RCTs (n=1445) found negligible effects of continuous passive motion (CPM) following total knee arthroplasty (TKA) on range of motion, pain, overall function, or quality of life for patients with osteoarthritis, and concluded that CPM does not have clinically important effects on recovery from TKA. E-SCOPE team members presented this evidence to the Regional Chiefs of Orthopedics, who agreed to reduce use. Initially, providers indicated that it was unlikely that CPM was still being used in Kaiser Permanente Southern California facilities. An assessment of data drawn from the EHR system, however, indicated that CPM was still in practice. To promote the deimplementation of CPM for TKA, recent TKA procedures were audited at the level of the ordering physician and reported to local clinical groups at each medical center. This report educates physicians, identifies high-utilizers of CPM, and is provided semi-annually to Chiefs of Orthopedic Surgery. When the project began in 2014, CPM was ordered for 12% of TKA procedures. In 2018, only 2.5% of TKAs included orders for CPM, a drop of over 60%.

Back to Top | Article Outline

Operational Interventions: Steam Sterilizer Shutdowns

In 2016, the E-SCOPE team identified a study21 which found that turning off sterilizers rather than using the “idle” setting was associated with substantial water and energy savings and did not result in adverse operational impacts. Stakeholder engagement began in early 2016 when the E-SCOPE team presented the evidence for steam sterilizer shutdown to Sterile Processing Directors and other stakeholders, including the Facilities Services Directors. Preliminary implementation efforts began in mid-2016.

Facilities Services Directors and Sterile Processing Directors staff were surveyed in 2016 and again in 2017 to determine whether steam sterilizers in their medical center areas could be shutdown, on what days and times eligible inactive sterilizer shutdowns would take place, and when the effort would begin. As of May 2018, 10 of 13 Kaiser Permanente Southern California facilities have reported successful shutdown of steam sterilizers during hours of nonuse (primarily on weekends, and some weeknights). Facilities Services estimates that current shutdown protocols could save between $250,000 and $300,000 per year regionwide in water and energy costs, along with other environmental benefits. Continued effort on this project has been transitioned to Facilities Management, with support from Kaiser Foundation Hospitals and Kaiser Health Plan Southern California. On the basis of successes in Southern California, 4 other Kaiser Permanente regions have adopted steam sterilizer shutdowns.

Back to Top | Article Outline

Summary: Factors Contributing to Accelerated Implementation

To accelerate implementation, a few elements have shown themselves to be crucial: (1) sponsorship from Senior Quality Executive(s), whose voice is respected in the organization and who understand the value added by a focus on EBM; (2) implementation project managers—2 full-time equivalents for over 7500 physicians—who provide steady/continuous project oversight to ensure that implementation plans are appropriately designed and executed; and (3) championship from local physician and staff leaders who socialize the new practices and facilitate implementation. E-SCOPE also takes advantage of regionwide internal communication channels, including clinical guideline and reference documents, education at local and regional clinical and operational meetings, and content provided through our EHR. Figure 3 illustrates the key facilitating factors used as implementation tools by our staff to help stakeholders ensure the effective implementation of selected practices.

Back to Top | Article Outline

CONCLUSIONS

The Kaiser Permanente Southern California E-SCOPE program was established in 2014 in an effort to reduce the knowledge to action gap in implementing up-to-date practices with a solid evidence base in clinical practice. The program makes use of EBM and implementation specialists to scan the current literature on a regular basis, to critically appraise the evidence to ensure that interventions can be moved into practice with confidence and without additional primary research, and to align implementation efforts with quality goals and standards. In 4 years, 30 interventions have been deployed, scaled up, or deimplemented within Southern California, and 5 have spread to other Kaiser Permanente regions, with a mean time from publication to implementation of 16 months.

Increasingly, implementation scientists are noting that a more comprehensive approach is needed to get the latest evidence into clinical practice.3 The tremendous growth in the number of studies published each year22 and their sometimes low quality makes it challenging, if not impossible, for health systems and clinicians to know which practices to implement. Working closely with stakeholders helps to overcome these barriers and to spread and reinforce a culture of evidence-based practice and ensures that implementation strategies are tailored to the intervention, setting, patient population, and available resources.

Damschroeder et al23 have described features of the setting in which implementation takes place that contributes to the likelihood that implementation efforts will be successful, many of which are characteristic of Kaiser Permanente Southern California. Foremost, Kaiser Permanente Southern California has a culture driven by a mission to be national leaders in health care quality, as well as a climate of the championship for EBM. E-SCOPE is only one part of a robust EBM Services Unit, which ensures that the care we provide and the guidelines we follow are based on the most current, highest-quality evidence available. There is a broad understanding of the need to grow as a learning health care system. Finally, leadership engagement and championship are probably the most important factors in creating practice change throughout the organization. However, there is considerable autonomy afforded at the level of the medical center and specialty group to accommodate local norms, values, and workflows.

One could express concern that E-SCOPE’s success may not be generalizable outside of a large, integrated health care delivery system. However, the characteristics above—effective leadership, the championship for quality, willingness to adapt to local needs—are true of many health care systems. We believe that, with modifications for local customs and structures, a program like E-SCOPE could be implemented successfully elsewhere. Where there are fewer resources to appraise the primary evidence, organizations could adjust the identification and screening process to their own needs, looking only, for instance, at practices supported by Cochrane reviews or those endorsed by the UK National Health Service or Australia’s Handbook of Non-Drug Interventions (HANDI), etc.

The program does have some limitations. First, at present, the program is limited only to the Southern California region. However, as E-SCOPE has grown, other regions within Kaiser Permanente, have expressed interest in the E-SCOPE approach. Efforts are ongoing to consider how best to help other Kaiser Permanente regions adopt the E-SCOPE process. Second, the E-SCOPE program relies heavily on input from a small but dedicated number of quality leaders to make decisions about which interventions to move forward as potential implementation projects. This “inside knowledge” is critical for understanding the system in which we practice, but the system is large—15 medical centers, 230 ambulatory clinics, and nearly 7500 providers across 48,000 square miles—making it difficult to know for sure what activities are taking place at each site. In 2019, 3 new physician quality leaders joined the team, all of whom have complementary knowledge about the system and the people who work in it, making it easier for us to tap into the right resources and identify the right stakeholders to more efficiently move projects forward.

E-SCOPE’s dedicated, centralized program for proactive identification and accelerated implementation, founded on stakeholder engagement and alignment of the most up-to-date scientific evidence with stakeholder priorities, has demonstrated that continuous learning and implementation can be efficiently promoted in a Learning Health Care System. Quality and clinical care leaders are encouraged to integrate the principles of the E-SCOPE approach into their own efforts to support evidence-based health care.

Back to Top | Article Outline

REFERENCES

1. Corlan D. Medline (PubMed) trend; 2012. Available at: http://dan.corlan.net/medline-trend.html. Accessed July 26, 2019.
2. Balas EA, Boren SA. Managing clinical knowledge for health care improvement. Yearb Med Inform. 2000;1:65–70.
3. Chambers CT. From evidence to influence: dissemination and implementation of scientific knowledge for improved pain research and management. Pain. 2018;159(suppl 1):S56–S64.
4. Balas EA, Chapman WW. Road map for diffusion of innovation in health care. Health Aff (Millwood). 2018;37:198–204.
5. Braithwaite J, Churruca K, Long JC, et al. When complexity science meets implementation science: a theoretical and empirical analysis of systems change. BMC Med. 2018;16:63.
6. Lynch EA, Chesworth BM, Connell LA. Implementation-the missing link in the research translation pipeline: is it any wonder no one ever implements evidence-based practice? Neurorehabil Neural Repair. 2018;32:751–761.
7. Kanter MH, Schottinger JE, Whittaker JL. A model for implementing evidence-based practices more quickly. NEJM Catalyst. 2017. Available at: https://catalyst.nejm.org/implementing-evidence-based-practices-quickly/.
8. Higgins JP, Altman DG, Gotzsche PC, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.
9. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336:924–926.
10. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.
11. Allanson ER, Tuncalp O, Vogel JP, et al. Implementation of effective practices in health facilities: a systematic review of cluster randomized trials. BMJ Glob Health. 2017;2:e000266.
12. Upala S, Sanguankeo A. Effect of lifestyle weight loss intervention on disease severity in patients with psoriasis: a systematic review and meta-analysis. Int J Obes (Lond). 2015;39:1197–1202.
13. Henry SL, Shen E, Ahuja A, et al. The online personal action plan: a tool to transform patient-enabled preventive and chronic care. Am J Prev Med. 2016;51:71–77.
14. AlFaleh K, Anabrees J. Probiotics for prevention of necrotizing enterocolitis in preterm infants. Cochrane Database Syst Rev. 2014;4:CD005496.
15. Athalye-Jape G, Rao S, Patole S. Lactobacillus reuteri DSM 17938 as a probiotic for preterm neonates: a strain-specific systematic review. JPEN J Parenter Enteral Nutr. 2016;40:783–794.
16. Athalye-Jape G, Deshpande G, Rao S, et al. Benefits of probiotics on enteral nutrition in preterm neonates: a systematic review. Am J Clin Nutr. 2014;100:1508–1519.
17. Rao SC, Athalye-Jape GK, Deshpande GC, et al. Probiotic supplementation and late-onset sepsis in preterm infants: a meta-analysis. Pediatrics. 2016;137:e20153684.
18. Olsen R, Greisen G, Schroder M, et al. Prophylactic probiotics for preterm infants: a systematic review and meta-analysis of observational studies. Neonatology. 2016;109:105–112.
19. Yang Y, Guo Y, Kan Q, et al. A meta-analysis of probiotics for preventing necrotizing enterocolitis in preterm neonates. Braz J Med Biol Res. 2014;47:804–810.
20. Harvey LA, Brosseau L, Herbert RD. Continuous passive motion following total knee arthroplasty in people with arthritis. Cochrane Database Syst Rev. 2014;2:CD004260.
21. McGain F, Moore G, Black J. Hospital steam sterilizer usage: could we switch off to save electricity and water? J Health Serv Res Policy. 2016;21:166–171.
22. van Rosmalen BV, Alldinger I, Cieslak KP, et al. Worldwide trends in volume and quality of published protocols of randomized controlled trials. PLoS ONE. 2017;12:e0173042.
23. Damschroder LJ, Aron DC, Keith RE, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.
Keywords:

evidence-based medicine; implementation; learning health care system

Copyright © 2019 Wolters Kluwer Health, Inc. All rights reserved.