Skip Navigation LinksHome > March 2014 - Volume 9 - Issue 2 > Searching with critical appraisal tools
Nursing Critical Care:
doi: 10.1097/01.CCN.0000444001.15811.a2

Searching with critical appraisal tools

Glasofer, Amy DrNP(c), MSN, RN, ONC

Free Access
Article Outline
Collapse Box

Author Information

Amy Glasofer is a senior educator for Virtua Center for Learning in Mt. Laurel, N.J.

CATs help journal club members rate the quality of research

The author has disclosed that she has no financial relationships related to this article.

Critical appraisal is “the process of assessing and interpreting evidence by systematically considering its validity, results and relevance to an individual's work.”1 Nurses rank feeling incapable of assessing the quality of research as one of the greatest barriers to using research in practice.2 Participation in a journal club (JC) can improve members' abilities to critically appraise the quality of research.3,4 The use of a formalized critical appraisal tool (CAT) during JC facilitates improvement in appraisal skills.3,4 The purpose of this article is to review the literature on selecting a CAT.

Figure. No caption a...
Image Tools
Back to Top | Article Outline

Literature review

CATs are designed to help readers rate the quality of research.5In reference to research, quality is the extent to which a study has minimized biases in the selection of subjects and measurement of outcomes, as well as minimized influence of anything outside of the factors being studied on the results.6 CATs are superior to informal appraisal in bringing readers with different levels and types of knowledge to a similar conclusion about a research paper.5 Their utility in a JC seems obvious; however, selecting a CAT isn't a simple task.

As a component of the Healthcare Research and Quality Act of 1999, Congress charged the Agency for Healthcare Research and Quality (AHRQ) with developing methods to ensure that reviews of healthcare literature are scientifically and clinically sound.6 To fulfill this charge, the AHRQ commissioned a study to describe systems that rate the quality of evidence and to provide guidance on best practices. This report analyzed over 120 published CATs.6 The AHRQ review, and others published since, have come to the same conclusions: there's no “gold standard” CAT for any specific study design, there's no generic tool that can be applied equally across study designs, and users of CATs should be careful about the CAT they select and how they use it.6–8

The AHRQ developed standard categories that any CAT should address to adequately rate specific research designs (see AHRQ critical rating categories for CATs by research design).6 Based on these criteria, they put forth 19 recommended CATs depending on research design (see AHRQ recommended CATs). Aside from being potentially outdated, the AHRQ-recommended CATs may be difficult to use in a nursing JC. Although different research designs require varying criteria for appraisal, it would be simpler to use CATs of similar formats across designs. Additionally, the CATs put forth by the AHRQ aren't inclusive of some forms of research and nonresearch evidence that JCs might wish to cover, including qualitative research, meta-synthesis, clinical practice guidelines, consensus or position statements, literature reviews, expert opinions, organizational experience, or case reports.28 Lastly, all of the CATs recommended by the AHRQ were developed for appraising medical literature. These tools may not translate easily for use in a nursing JC.

Back to Top | Article Outline

Selecting our CAT

For these reasons, the JC at our institution opted to evaluate additional CATs. One of the JC facilitators who was completing her doctoral coursework took on this task. Several resources were utilized including searching academic databases and the Internet, and consulting with mentors and evidence-based practice texts. Ultimately, the CATs selected for our JC came from the Johns Hopkins Model for Evidence Based Practice.28 This reference contains two tools for appraising individual articles. The first is for appraising research evidence (randomized control trials, quasi-experimental studies, nonexperimental studies, qualitative studies, systematic reviews, and systematic reviews with meta-analysis or meta-synthesis). All of the categories considered critical for systematic reviews, randomized control trials, and observational studies by the AHRQ are covered in this CAT, with the exception of assessing for funding or sponsorship. The AHRQ allowed for the absence of funding criteria in recommended CATs as this category was so often not addressed.6 The second Johns Hopkins CAT is for evaluating nonresearch articles.28 There's less guidance available for evaluating a nonresearch CAT. However, the Johns Hopkins tool is based on established criteria for appraising nonresearch evidence.28–30

There are some limitations to the Johns Hopkins CATs. First, the research appraisal tool applies a single set of questions across multiple research designs and depends on the user to determine if the question is appropriate. The AHRQ cautioned that utilizing such a generalized tool could weaken the analysis. Although our JC had received education on critical appraisal and the use of the tools, members did struggle initially with determining which questions applied to the study they were critiquing. As they grew more comfortable with the CAT, with various research designs, and with research terminology, JC members did become more fluent with utilizing the tool. Additionally, the research appraisal tool doesn't contain the AHRQ-recommended criteria for evaluating research on a diagnostic study, such as the reliability and validity of a new blood test. This hasn't yet been an issue for our JC because the group hasn't selected any diagnostic studies for review. We would have to choose a different CAT should this ever occur. Finally, when our JC formed, we weren't utilizing the most current version of the Johns Hopkins CATs.31 These older versions rely on the user to determine if the study is a research study versus a quality improvement project, for example. At the onset of our JC, members had difficulty in naming the study design. However, the CATs from the second edition include an algorithm to assist the user in defining the research design and selecting the appropriate CAT.28 This feature would be very helpful to novice JC members.

Table AHRQ critical ...
Table AHRQ critical ...
Image Tools
Table AHRQ recommend...
Table AHRQ recommend...
Image Tools

Aside from these few limitations, the Johns Hopkins CATs have proven to be excellent tools for critical appraisal in our JC. Having only two forms to choose from, members were quickly able to select the appropriate CAT for each study and readily grew accustomed to the tool formats. It also eased the logistics of ensuring that each participant had the necessary forms. Though conversation in JC initially focused on summaries of the article and whether participants liked the article or not, content shifted to truly being an analysis of the quality of the research and its applicability to practice as members became more skilled in critical appraisal. Surveys of participants' perceptions of barriers to research utilization were conducted at baseline, at 6 months, and at 2 years after initiation of the JC.33 Participant perception of their own research values, skills, and awareness as barriers to research utilization decreased by 18% at 6 months, and 22% after the JC had been established for 2 years. During an unrelated meeting regarding a practice change, one clinical nurse remarked that she had recently read a randomized controlled trial on the topic, and that she actually understood what that meant thanks to JC. Participation in a JC, utilizing a formalized CAT, can help nurses feel more capable of assessing the quality of research, an important step in promoting the use of research in practice.

Back to Top | Article Outline


1. Horsley T, Hyde C, Santesso N, Parkes J, Milne R, Stewart R. Teaching critical appraisal skills in healthcare settings. Cochrane Database Syst Rev. 2011;(11):CD001270.

2. Kajermo KN, Boström AM, Thompson DS, Hutchinson AM, Estabrooks CA, Wallin L. The BARRIERS Scale — the barriers to research utilization scale: a systematic review. Implement Sci. 2010;5:32.

3. Glasofer A. The role of journal clubs in bridging the research-practice gap: a review of the nursing literature. Nursing2013 Critical Care. 2013;8(4):41–45.

4. Harris J, Kearley K, Heneghan C, et al. Are journal clubs effective in supporting evidence-based decision making? A systematic review. BEME Guide No. 16. Med Teach. 2011;33(1):9–23.

5. Crowe M, Sheppard L, Campbell A. Comparison of the effects of using the Crowe Critical Appraisal Tool versus informal appraisal in assessing health research: a randomised trial. Int J Evid Based Healthc. 2011;9(4):444–449.

6. West S, King V, Carey TS, et al. Systems to Rate the Strength of Scientific Evidence. Evidence Report/Technology Assessment No. 47 (Prepared by the Research Triangle Institute-University of North Carolina Evidence-based Practice Center under Contract No. 290-97-0011). AHRQ Publication No. 02-E016. Rockville, MD: Agency for Healthcare Research and Quality; 2002.

7. Crowe M, Sheppard L. A review of critical appraisal tools show they lack rigor: alternative tool structure is proposed. J Clin Epidemiol. 2011;64(1):79–89.

8. Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA. A systematic review of the content of critical appraisal tools. BMC Med Res Methodol. 2004;4:22.

9. Barnes DE, Bero LA. Why review articles on the health effects of passive smoking reach different conclusions. JAMA. 1998;279(19):1566–1570.

10. Irwig L, Tosteson AN, Gatsonis C, et al. Guidelines for meta-analyses evaluating diagnostic tests. Ann Intern Med. 1994;120(8):667–676.

11. Khan KS, Ter Riet G, Glanville J, Sowden AJ, Kleijnen J. Undertaking Systematic Reviews of Research on Effectiveness. CRD's Guidance for Carrying Out or Commissioning Reviews. York, England: University of York, NHS Centre for Reviews and Dissemination; 2000.

12. Sacks HS, Reitman D, Pagano D, Kupelnick B. Meta-analysis: an update. Mt Sinai J Med. 1996;63(3-4):216–224.

13. Chalmers TC, Smith H Jr, Blackburn B, et al. A method for assessing the quality of a randomized control trial. Control Clin Trials. 1981;2(1):31–49.

14. de Vet HCW de Bie RA, van der Heijden GJMG, Verhagen AP, Sijpkes P, Kipschild PG. Systematic reviews on the basis of methodological criteria. Physiotherapy. 1997. 83(6):284–289.

15. Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health. 1998;52(6):377–384.

16. Harbour R, Miller J. A new system for grading recommendations in evidence based guidelines. BMJ. 2001;323(7308):334–336.

17. Liberati A, Himel HN, Chalmers TC. A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol. 1986;4(6):942–951.

18. Reisch JS, Tyson JE, Mize SG. Aid to the evaluation of therapeutic studies. Pediatrics. 1989;84(5):815–827.

19. Sindhu F, Carpenter L, Seers K. Development of a tool to rate the quality assessment of randomized controlled trials using a Delphi technique. J Adv Nurs. 1997;25(6):1262–1268.

20. van der Heijden GJ, van der Windt DA, Kleijnen J, Koes BW, Bouter LM. Steroid injections for shoulder disorders: a systematic review of randomized clinical trials. Br J Gen Pract. 1996;46(406):309–316.

21. Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med. 1994;121(1):11–21.

22. Spitzer WO, Lawrence V, Dales R, et al. Links between passive smoking and disease: a best-evidence synthesis. A report of the Working Group on Passive Smoking. Clin Invest Med. 1990;13(1):17-42; discussion 43–46.

23. Zaza S, Wright-De Agüero LK, Briss PA, et al. Data collection instrument and procedure for systematic reviews in the Guide to Community Preventive Services. Task Force on Community Preventive Services. Am J Prev Med. 2000;18(1 suppl):44–74.

24. Cochrane Methods Working Group on Systematic Reviews of Screening and Diagnostic Tests Recommended Methods; 1996.

25. Lijmer JG, Mol BW, Heisterkamp S, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999;282(11):1061–1066.

26. Dearholt SL, Dang D. Johns Hopkins Nursing Evidence-Based Practice: Model and Guidelines. Second Edition. Indianapolis, IN: Sigma Theta Tau International; 2012.

27. Carande-Kulis VG, Maciosek MV, Briss PA, et al. Methods for systematic reviews of economic evaluations for the Guide to Community Preventive Services. Task Force on Community Preventive Services. Am J Prev Med. 2000;18(1 suppl):75–91.

28. Fervers B, Burgers JS, Haugh MC, et al. Predictors of high quality clinical practice guidelines: examples in oncology. Int J Qual Health Care. 2005;17(2):123–132.

29. Newhouse RP, Dearholt SL, Poe SS, Pugh LC, White KM. Johns Hopkins Nursing Evidence-Based Practice Research Evidence Appraisal.

30. Newhouse RP, Dearholt SL, Poe SS, Pugh LC, White KM. Johns Hopkins Nursing Evidence-Based Practice Non-Research Evidence Appraisal.

31. Funk SG, Champagne MT, Wiese RA, Tornquist EM. BARRIERS: the barriers to research utilization scale. Appl Nurs Res. 1991;4(1):39–45.

© 2014 by Lippincott Williams & Wilkins, Inc.