Evidence-Based Practice, Step by Step: Critical Appraisal of the Evidence Part III

Abstract

The process of synthesis: seeing similarities and differences across the body of evidence.

This is the seventh article in a series from the Arizona State University College of Nursing and Health Innovation's Center for the Advancement of Evidence-Based Practice. Evidence-based practice (EBP) is a problem-solving approach to the delivery of health care that integrates the best evidence from studies and patient care data with clinician expertise and patient preferences and values. When delivered in a context of caring and in a supportive organizational culture, the highest quality of care and best patient outcomes can be achieved.

The purpose of this series is to give nurses the knowledge and skills they need to implement EBP consistently, one step at a time. Articles will appear every two months to allow you time to incorporate information as you work toward implementing EBP at your institution. Also, we've scheduled "Chat with the Authors" calls every few months to provide a direct line to the experts to help you resolve questions. See details below.

In Brief

In their seventh article on implementing evidence-based practice, the authors begin the process of evaluation and synthesis of the evidence.

Box: Need Help with Evidence-Based Practice? Chat with the Authors on November 16!

In September's evidence-based practice (EBP) article, Rebecca R., our hypothetical staff nurse, Carlos A., her hospital's expert EBP mentor, and Chen M., Rebecca's nurse colleague, rapidly critically appraised the 15 articles they found to answer their clinical question—"In hospitalized adults (P), how does a rapid response team (I) compared with no rapid response team (C) affect the number of cardiac arrests (O) and unplanned admissions to the ICU (O) during a three-month period (T)?"—and determined that they were all "keepers." The team now begins the process of evaluation and synthesis of the articles to see what the evidence says about initiating a rapid response team (RRT) in their hospital. Carlos reminds them that evaluation and synthesis are synergistic processes and don't necessarily happen one after the other. Nevertheless, to help them learn, he will guide them through the EBP process one step at a time.

Table 1: Final Evaluation Table

STARTING THE EVALUATION

Rebecca, Carlos, and Chen begin to work with the evaluation table they created earlier in this process when they found and filled in the essential elements of the 15 studies and projects (see "Critical Appraisal of the Evidence: Part I," July). Now each takes a stack of the "keeper" studies and systematically begins adding to the table any remaining data that best reflect the study elements pertaining to the group's clinical question (see Table 1; for the entire table with all 15 articles, go to https://links.lww.com/AJN/A17). They had agreed that a "Notes" section within the "Appraisal: Worth to Practice" column would be a good place to record the nuances of an article, their impressions of it, as well as any tips—such as what worked in calling an RRT—that could be used later when they write up their ideas for initiating an RRT at their hospital, if the evidence points in that direction. Chen remarks that although she thought their initial table contained a lot of information, this final version is more thorough by far. She appreciates the opportunity to go back and confirm her original understanding of the study essentials.

The team members discuss the evolving patterns as they complete the table. The three systematic reviews, which are higher-level evidence, seem to have an inherent bias in that they included only studies with control groups. In general, these studies weren't in favor of initiating an RRT. Carlos asks Rebecca and Chen whether, now that they've appraised all the evidence about RRTs, they're confident in their decision to include all the studies and projects (including the lower-level evidence) among the "keepers." The nurses reply with an emphatic affirmative! They tell Carlos that the projects and descriptive studies were what brought the issue to life for them. They realize that the higher-level evidence is somewhat in conflict with the lower-level evidence, but they're most interested in the conclusions that can be drawn from considering the entire body of evidence.

Rebecca and Chen admit they have issues with the systematic reviews, all of which include the MERIT study.^1-4 In particular, they discuss how the authors of the systematic reviews made sure to report the MERIT study's finding that the RRT had no effect, but didn't emphasize the MERIT study authors' discussion about how their study methods may have influenced the reliability of the findings (for more, see "Critical Appraisal of the Evidence: Part II," September). Carlos says that this is an excellent observation. He also reminds the team that clinicians may read a systematic review for the conclusion and never consider the original studies. He encourages Rebecca and Chen in their efforts to appraise the MERIT study and comments on how well they're putting the pieces of the evidence puzzle together. The nurses are excited that they're able to use their new knowledge to shed light on the study. They discuss with Carlos how the interpretation of the MERIT study has perhaps contributed to a misunderstanding of the impact of RRTs.

Comparing the evidence. As the team enters the lower-level evidence into the evaluation table, they note that it's challenging to compare the project reports with studies that have clearly described methodology, measurement, analysis, and findings. Chen remarks that she wishes researchers and clinicians would write study and project reports similarly. Although each of the studies has a process or method determining how it was conducted, as well as how outcomes were measured, data were analyzed, and results interpreted, comparing the studies as they're currently written adds another layer of complexity to the evaluation. Carlos says that while it would be great to have studies and projects written in a similar format so they're easier to compare, that's unlikely to happen. But he tells the team not to lose all hope, as a format has been developed for reporting quality improvement initiatives called the SQUIRE Guidelines; however, they aren't ideal. The team looks up the guidelines online (www.squire-statement.org) and finds that the Institute for Healthcare Improvement (IHI) as well as a good number of journals have encouraged their use. When they review the actual guidelines, the team notices that they seem to be focused on research; for example, they require a research question and refer to the study of an intervention, whereas EBP projects have PICOT questions and apply evidence to practice. The team discusses that these guidelines can be confusing to the clinicians authoring the reports on their projects. In addition, they note that there's no mention of the synthesis of the body of evidence that should drive an evidence-based project. While the SQUIRE Guidelines are a step in the right direction for the future, Carlos, Rebecca, and Chen conclude that, for now, they'll need to learn to read these studies as they find them—looking carefully for the details that inform their clinical question.

Once the data have been entered into the table, Carlos suggests that they take each column, one by one, and note the similarities and differences across the studies and projects. After they've briefly looked over the columns, he asks the team which ones they think they should focus on to answer their question. Rebecca and Chen choose "Design/Method," "Sample/Setting," "Findings," and "Appraisal: Worth to Practice" (see Table 1) as the initial ones to consider. Carlos agrees that these are the columns in which they're most likely to find the most pertinent information for their synthesis.

SYNTHESIZING: MAKING DECISIONS BASED ON THE EVIDENCE

Design/Method. The team starts with the "Design/Method" column because Carlos reminds them that it's important to note each study's level of evidence. He suggests that they take this information and create a synthesis table (one in which data is extracted from the evaluation table to better see the similarities and differences between studies) (see Table 2^1-15). The synthesis table makes it clear that there is less higher-level and more lower-level evidence, which will impact the reliability of the overall findings. As the team noted, the higher-level evidence is not without methodological issues, which will increase the challenge of coming to a conclusion about the impact of an RRT on the outcomes.

Sample/Setting. In reviewing the "Sample/Setting" column, the group notes that the number of hospital beds ranged from 218 to 662 across the studies. There were several types of hospitals represented (4 teaching, 4 community, 4 no mention, 2 acute care hospitals, and 1 public hospital). The evidence they've collected seems applicable, since their hospital is a community hospital.

Findings. To help the team better discuss the evidence, Carlos suggests that they refer to all projects or studies as "the body of evidence." They don't want to get confused by calling them all studies, as they aren't, but at the same time continually referring to "studies and projects" is cumbersome. He goes on to say that, as part of the synthesis process, it's important for the group to determine the overall impact of the intervention across the body of evidence. He helps them create a second synthesis table containing the findings of each study or project (see Table 3^1-15). As they look over the results, Rebecca and Chen note that RRTs reduce code rates, particularly outside the ICU, whereas unplanned ICU admissions (UICUA) don't seem to be as affected by them. However, 10 of the 15 studies and projects reviewed didn't evaluate this outcome, so it may not be fair to write it off just yet.

The EBP team can tell from reading the evidence that researchers consider the impact of an RRT on hospital-wide mortality rates (HMR) as the more important outcome; however, the group remains unconvinced that this outcome is the best for evaluating the purpose of an RRT, which, according to the IHI, is early intervention in patients who are unstable or at risk for cardiac or respiratory arrest.¹⁶ That said, of the 11 studies and projects that evaluated mortality, more than half found that an RRT reduced it. Carlos reminds the group that four of those six articles are level-VI evidence and that some weren't research. The findings produced at this level of evidence are typically less reliable than those at higher levels of evidence; however, Carlos notes that two articles having level-VI evidence, a study and a project, had statistically significant (less likely to occur by chance, P < 0.05) reductions in HMR, which increases the reliability of the results.

Chen asks, since four level-VI reports documented that an RRT reduces HMR, should they put more confidence in findings that occur more than once? Carlos replies that it's not the number of studies or projects that determines the reliability of their findings, but the uniformity and quality of their methods. He recites something he heard in his Expert EBP Mentor program that helped to clarify the concept of making decisions based on the evidence: the level of the evidence (the design) plus the quality of the evidence (the validity of the methods) equals the strength of the evidence, which is what leads clinicians to act in confidence and apply the evidence (or not) to their practice and expect similar findings (outcomes). In terms of making a decision about whether or not to initiate an RRT, Carlos says that their evidence stacks up: first, the MERIT study's results are questionable because of problems with the study methods, and this affects the reliability of the three systematic reviews as well as the MERIT study itself; second, the reasonably conducted lower-level studies/projects, with their statistically significant findings, are persuasive. Therefore, the team begins to consider the possibility that initiating an RRT may reduce code rates outside the ICU (CRO) and may impact non-ICU mortality; both are outcomes they would like to address. The evidence doesn't provide equally promising results for UICUA, but the team agrees to include it in the outcomes for their RRT project because it wasn't evaluated in most of the articles they appraised.

As the EBP team continues to discusses probable outcomes, Rebecca points to one study's data in the "Findings" column that shows a financial return on investment for an RRT.⁹ Carlos remarks to the group that this is only one study, and that they'll need to make sure to collect data on the costs of their RRT as well as the cost implications of the outcomes. They determine that the important outcomes to measure are: CRO, non-ICU mortality (excluding patients with do not resuscitate [DNR] orders), UICUA, and cost.

Appraisal: Worth to Practice. As the team discusses their synthesis and the decision they'll make based on the evidence, Rebecca raises a question that's been on her mind. She reminds them that in the "Appraisal: Worth to Practice" column, teaching was identified as an important factor in initiating an RRT and expresses concern that their hospital is not an academic medical center. Chen reminds her that even though theirs is not a designated teaching hospital with residents on staff 24 hours a day, it has a culture of teaching that should enhance the success of an RRT. She adds that she's already hearing a buzz of excitement about their project, that their colleagues across all disciplines have been eager to hear the results of their review of the evidence. In addition, Carlos says that many resources in their hospital will be available to help them get started with their project and reminds them of their hospital administrators' commitment to support the team.

Table 2: The 15 Studies: Levels and Types of Evidence

Table 3: Effect of the Rapid Response Team on Outcomes

ACTING ON THE EVIDENCE

As they consider the synthesis of the evidence, the team agrees that an RRT is a valuable intervention to initiate. They decide to take the criteria for activating an RRT from several successful studies/projects and put them into a synthesis table to better see their major similarities (see Table 4^4,8,9,13,15). From this combined list, they choose the criteria for initiating an RRT consult that they'll use in their project (see Table 5). The team also begins discussing the ideal make up for their RRT. Again, they go back to the evaluation table and look over the "Major Variables Studied" column, noting that the composition of the RRT varied among the studies/projects. Some RRTs had active physician participation (n = 6), some had designated physician consultation on an as-needed basis (n = 2), and some were nurse-led teams (n = 4). Most RRTs also had a respiratory therapist (RT). All RRT members had expertise in intensive care and many were certified in advanced cardiac life support (ACLS). They agree that their team will be comprised of ACLS-certified members. It will be led by an acute care nurse practitioner (ACNP) credentialed for advanced procedures, such as central line insertion. Members will include an ICU RN and an RT who can intubate. They also discuss having physicians willing to be called when needed. Although no studies or projects had a chaplain on their RRT, Chen says that it would make sense in their hospital. Carlos, who's been on staff the longest of the three, says that interdisciplinary collaboration has been a mainstay of their organization. A physician, ACNP, ICU RN, RT, and chaplain are logical choices for their RRT.

As the team ponders the evidence, they begin to discuss the next step, which is to develop ideas for writing their project implementation plan (also called a protocol). Included in this protocol will be an educational plan to let those involved in the project know information such as the evidence that led to the project, how to call an RRT, and outcome measures that will indicate whether or not the implementation of the evidence was successful. They'll also need an evaluation plan. From reviewing the studies and projects, they also realize that it's important to focus their plan on evidence implementation, including carefully evaluating both the process of implementation and project outcomes.

Be sure to join the EBP team in the next installment of this series as they develop their implementation plan for initiating an RRT in their hospital, including the submission of their project proposal to the ethics review board.

Table 4: Defined Criteria for Initiating an RRT Consult

Table 5: Defined Criteria for Initiating an RRT Consult at Our Hospital