- The literature review is up-to-date.
- The number of references is appropriate and their selection is judicious.
- The review of the literature is well integrated.
- The references are mainly primary sources.
- Ideas are acknowledged appropriately (scholarly attribution) and accurately.
- The literature is analyzed and critically appraised.
ISSUES AND EXAMPLES RELATED TO THE CRITERIA
Research questions come from observing phenomena or reading the literature. Regardless of what inspired the research, however, study investigators must thoroughly review existing literature to adequately understand the scope of the issues relevant to their questions. Although systematic reviews of the literature conducted in the social and biomedical sciences, such as those produced by the Cochrane Collaboration (for clinical issues) and the Campbell Collaboration (for areas of social science) may be quite different in terms of the types of evidence provided and the natures of the outcomes, their goals are the same, that is, to present the best evidence to inform research, practice, and policy. These reviews are usually carried out by large teams, which follow strict protocols common to the whole collaboration. Individual researchers also conduct thorough reviews, albeit usually less structured and in-depth. They achieve three key research aims through a thorough analysis of the literature: refinement of their research questions, defense of their research design, and ultimately support of their interpretations of outcomes and conclusions. Thus, in the research report, the reviewer should find a clear demonstration of the literature's contribution to the study and its context.1
Before discussing the specifics of each of the three aims, it is important to offer some distinctions regarding the research continuum. Where researchers fit along the quantitative—qualitative continuum influences how they use literature within a study, although there are no rigid rules about how to use it. Typically, at the quantitative end of the spectrum, researchers review the bulk of the literature primarily at the beginning of the study in order to establish the theoretical or conceptual framework for the research question or problem. They also use the literature to validate the use of specific methods, tools, and (statistical) analyses, adding citations in the appropriate sections of the manuscript. At the qualitative end of the spectrum, the researchers weave the relevant literature into all phases of the study and use it to guide the evolution of their thinking as data are gathered, transcribed, excerpted, analyzed, and placed before the reader.2 They also use the literature to reframe the problem as the study evolves. Although the distinction is not crystal-clear, the difference between the ends of the continuum might be viewed as the difference between testing theory-driven hypotheses (quantitative) and generating theory-building hypotheses (qualitative).
Researchers all along this continuum use the literature to inform their early development of research interests, problems, and questions and later in the conduct of their research and the interpretation of their findings. A review of relevant literature sets the stage for a study. It provides a logically organized world view of the researcher's question, or of the situation the researcher has observed—what knowledge exists relevant to the research question, how the question or problem has been previously studied (types of designs and methodologic concerns), and the concepts and variables that have been shown to be associated with the problem (question).3 The researcher evaluates previous work in terms of its relevance to the research question of interest,4 and synthesizes what is known, noting relationships that have been well studied and identifying areas for elaboration, questions that remain unanswered, or gaps in understanding.1,3,5,6 The researcher documents the history and present status of the study's question or problem. The literature reviewed should not only be current, but also reflect the contributions of salient published and unpublished research, which may be quite dated but play a significant role in the evolution of the research. Regardless of perspective (qualitative, quantitative, or mixed method), the researcher must frame the problem or research questions as precisely as possible from a chronologic and developmental perspective, given the confines of the literature.2 For example, when presenting the tenets of adult learning as a basis for program evaluation an author would be remiss if he or she omitted the foundational writings of Knowles,7 Houle,8 and perhaps Lindeman9 from the discussion.
Equally important to using the literature to identify current knowledge is using it to defend and support the study and to inform the design and methods.10 The researcher interprets and weighs the evidence, presents valid points making connections between the literature and the study design, reasons logically for specific methods, and describes in detail the variables or concepts that will be scrutinized. Through the literature, the researcher provides a map guiding the reader to the conclusion that the current study is important and necessary and the design is appropriate to answer the questions.6
Once they have the study outcomes, researchers offer explanations, challenge assumptions, and make recommendations considering the literature used initially to frame the research problem. Authors may apply some of the most salient literature at the end of the manuscript to support their conclusions (fully or partially), refute current knowledge, revise a hypothesis, or reframe the problem.5 The authors use literature to bring the reader back to the theory tested (quantitative) or the theory generated (qualitative).
Reviewers must consider the pertinence of the literature and documentation with regard to the three key research aims stated earlier. They should also consider the types of resources cited and the balance of the perspectives discussed within the literature reviewed. When considering the types of resources cited, reviewers should determine whether the references are predominantly general sources (textbooks),4 primary sources (research articles written by those who conducted the research),4 or secondary sources (articles where a researcher describes the work of others).4 References should be predominantly primary sources, whether published or unpublished. Secondary sources are acceptable, and desirable, if primary sources are unavailable or if they provide a review (meta-analysis, for example) of what is known about the research problem. Researchers may use general resources as a basis for describing, for example, a theoretical or methodologic principle, or a statistical procedure.
Researchers may have difficulty finding all of the pertinent literature because it may not be published (dissertations), and not all published literature is indexed in electronic databases. Manual searching is still necessary. Reviewers are cautioned to look for references that appear inclusive of the whole body of existing literature. For example, some relevant articles are not indexed in Medline, but are indexed in ERIC. Reviewers can tell whether multiple databases were searched for relevant literature by the breadth of disciplines represented by the citations. Thus, it is important that the researcher describe how he or she found the previous work used to study his or her problem.11
A caveat for reviewers is to be wary of researchers who have not carried out a thorough review of the literature. They may report that there is a paucity of research in their area when in fact plenty exists. At times, authors must be pushed. At the very minimum, reviewers should comment on whether the researchers described to the reviewers' satisfaction how they found study-related literature and the criteria used to select the sources that were discussed. Reviewers must decide whether this process was satisfactorily described. If only published reports found in electronic databases are discussed, then the viewpoint presented “may be biased toward well-known research” that presents only statistically significant outcomes.1
When considering the perspectives presented by the author, reviewers should pay attention to whether the discussion presents all views that exist in the literature base, that is, conflicting, consensus, or controversial opinions.5,12 The thoroughness of the discussion also depends upon the author's explanation of how literature was located and chosen for inclusion. For example, Bland and colleagues13 have provided an excellent example of how the process of location and selection was accomplished.
1. Haller KB. Conducting a literature review. MCN: Am Maternal Child Nurs. 1988;13:148.
2. Haller EJ, Kleine PF. Teacher empowerment and qualitative research. In: Haller EJ, Kleine PF (eds). Using Educational Research: A School Administrator's Guide. New York: Addison Wesley Longman, 2000:193–237.
3. Rodgers J, Smith T, Chick N, Crisp J. Publishing workshops number 4. Preparing a manuscript: reviewing literature. Nursing Praxis in New Zealand. 1997;12:38–42.
4. Fraenkel JR, Wallen NE. How to Design and Evaluate Research in Education. 4th ed. Boston, MA: McGraw-Hill Higher Education, 2000.
5. Martin PA. Writing a useful literature review for a quantitative research project. Appl Nurs Res. 1997;10:159–62.
6. Bartz C. It all starts with an idea. Alzheimer Dis and Assoc Dis. 1999;13:S106–S110.
7. Knowles MS. The Modern Practice of Adult Education: From Pedagogy to Andragogy. Chicago, IL: Association Press, 1980.
8. Houle CO. The Inquiring Mind. Madison, WI: University of Wisconsin Press, 1961.
9. Lindeman EC. The Meaning of Adult Education. Norman, OK: University of Oklahoma Research Center for Continuing Professional and Higher Education, 1989 [Orig. pub. 1926].
10. Glesne C, Peshkin A. Becoming Qualitative Researchers: An Introduction. White Plains, NY: Longman Publishing Group, 1992.
11. Smith AJ, Goodman NW. The hypertensive response to intubation: do researchers acknowledge previous work? Can J Anaesth. 1997;44:9–13.
12. Bruette V, Fitzig C. The literature review. J NY State Nurs Assoc. 1993;24:14–5.
13. Bland CJ, Meurer LN, Maldonado G. Determinants of primary care specialty choice: a non-statistical meta-analysis of the literature. Acad Med. 1995;70:620–41.
Bartz C. It all starts with an idea. Alzheimer Dis and Assoc Dis. 1999;13:S106-S110.
Best Evidence in Medical Education (BEME). 〈http://www.mailbase.ac.uk/lists/beme
〉. Accessed 3/30/01.
Bland CJ, Meurer LN, Maldonado G. A systematic approach to conducting a non-statistical meta-analysis of research literature. Acad Med. 1995;70:642–53.
The Campbell Collaboration. 〈http://campbell.gse.upenn.edu
〉. Accessed 3/30/01.
Review Criteria for Research Manuscripts
Joint Task Force of Academic Medicine and the GEA-RIME Committee