- The study is relevant to the mission of the journal or its audience.
- The study addresses important problems or issues; the study is worth doing.
- The study adds to the literature already available on the subject.
- The study has generalizability because of the selection of subjects, setting, and educational intervention or materials.
ISSUES AND EXAMPLES RELATED TO CRITERIA
An important consideration for editors in deciding whether to publish an article is its relevance to the community (or usually, communities) the journal serves. Relevance has several connotations and all are judged with reference to a specific group of professionals and to the tasks of that group. Indeed, one thing is often spoken of as being “relevant to” something else, and that something is the necessary context that establishes relevance.
First, editors and reviewers must gauge the applicability of the manuscript to problems within the journal's focus; the more common or important the problem addressed by an article is to those involved in it, the more relevant it is. The essential issue is whether a rigorous answer to this study's question will affect what readers will do in their daily work, for example, or what researchers will do in their next study, or even what policymakers may decide. This can be true even if a study is “negative,” that is, does not confirm the hypothesis at hand. For studies without hypotheses (for instance, a systematic review of prior research or a meta-analysis), the same question applies: Does this review achieve a synthesis that will directly affect what readers do?
Second, a manuscript, especially one involving qualitative research, may be pertinent to the community by virtue of its contribution to theory building, generation of new hypotheses, or development of methods. In this sense, the manuscript introduces, refines, or critiques issues that, for example, underlie the teaching and practice of medicine, such as cognitive psychology, ethics, and epistemology. Thus a study may be quite relevant even though its immediate, practical application is not worked out.
Third, each manuscript must be judged with respect to its appropriateness to the mission of the specific journal. Reviewers should consider these three elements of relevance irrespective of the merit or quality of an article.
The relevance of an article is often most immediately apparent in the first paragraphs of a manuscript, especially in how the research question or problem posed by the paper is framed. As discussed earlier in “Problem Statement, Conceptual Framework, and Research Question,” an effective article explicitly states the issue to be addressed, in the form of either a question to be answered or a controversy to be resolved. A conceptual or theoretical framework underlies a research question, and a manuscript is stronger when this framework is made explicit. An explicit presentation of the conceptual framework helps the reviewer and makes the study's importance or relevance more clear.
The relevance of a research manuscript may be gauged by its purpose or the intention of the study, and a vocabulary drawn from clinical research is quite applicable here. Feinstein classifies research according to its “architecture,” the effort to create and evaluate research structures that have both “the reproducible documentation of science and the elegant design of art.”1 Descriptive research provides collections of data that characterize a problem or provide information; no comparisons are inherent in the study design, and the observations may be used for policy decisions or to prepare future, more rigorous studies. Many papers in social science journals, including those in health professions education, derive their relevance from such an approach. In cause—effect research, specific comparisons (for instance, to the subjects' baseline status or to a separate control group) are made to reach conclusions about the efficacy or impact of an intervention (for instance, a new public health campaign or an innovative curriculum). The relevance of such research architecture derives from its power to establish the causality, or at least the strong effects, from innovations. In research that deals with process issues, as defined by Feinstein, the products of a new process or the performance of a particular procedure (for instance, a new tool for the assessment of clinical competence) are studied as an indication of the quality or value of the process or procedure. In this case relevance is not from a cause-and-effect relationship but from a new measurement tool that could be applied to a wide variety of educational settings.1, p.15–16
The relevance of a topic is related to, but is not the same as, the feasibility of answering a research question. Feasibility is related to study design and deals with whether and how we can get an answer. Relevance more directly addresses whether the question is significant enough to be worth asking.2 The relevance of a manuscript is more complex than that of the topic per se, and the relevance includes the importance of the topic as well as whether the execution of the study or of the discussion is powerful enough to affect what others in the field think or do.
Relevance is, at times, a dichotomous, or “yes—no,” decision; but often it is a matter of degree, as illustrated by the criteria. In this more common circumstance, relevance is a summary conclusion rather than a simple observation. It is a judgment supported by the applicability of the principles, methods, instruments, and findings that together determine the weight of the relevance. Given a limited amount of space in each issue of a journal, editors have to choose among competing manuscripts, and relevance is one way of summarizing the importance of a manuscript's subject, thesis, and conclusions to the journal's readership.
Certain characteristics or strengths can establish a manuscript's relevance: Would a large part of the journal's community—or parts of several of its overlapping communities—consider the paper worth reading? Is it important that this paper be published even though the journal can publish only a fixed percentage of the manuscripts it receives each year? As part of their recommendation to the editor (see Chapter 3), reviewers are asked to rate how important a manuscript is: extremely, very, moderately, slightly, or not important. Issues that may influence reviewers and editors to judge a paper to be relevant include:
- Irrespective of a paper's methods or study design, the topic at hand would be considered common and/or serious by the readership. As stated before, relevance is a summary judgment and not infallible. One study of clinical research papers showed that readers did not always agree with reviewers on the relevance of studies to their own practice.3 Editors of medical education research journals, for example, must carefully choose to include the perspective of educational practitioners in their judgment of relevance, and try to reflect the concerns of these readers.
- Irrespective of immediate and practical application, the author(s) provides important insights for understanding theory, or the paper suggests innovations that have the potential to advance the field. In this respect, a journal leads its readership and does not simply reflect it. The field of clinical medicine is filled with examples of great innovations, such as the initial description of radioimmunoassay or the citric acid cycle by Krebs, that were initially rejected for publication.4 To use medical education as the example again, specific evaluation methods, such as using actors to simulate patients, gradually pervaded undergraduate medical education but initially might have seemed unfeasible.5
- The methods or conclusions described in the paper are applicable in a wide variety of settings.
In summary, relevance is a necessary but not sufficient criterion for the selection of articles to publish in journals. The rigorous study of a trivial problem, or one already well studied, would not earn pages in a journal that must deal with competing submissions. Reviewers and editors must decide whether the question asked is worth answering at all, whether its solution will contribute, immediately or in the longer term, to the work of medical education and, finally, whether the manuscript at hand will be applicable to the journal's readership.
1. Feinstein AR. Clinical Epidemiology: The Architecture of Clinical Research. Philadelphia, PA: W. B. Saunders, 1985;4.
2. Fraenkel JR, Wallen NE. How to Design and Evaluate Research in Education. 4th ed. New York: McGraw—Hill Higher Education, 2000:30–7.
3. Justice AC, Berlin JA, Fletcher SW, Fletcher RH, Goodman SN. Do readers and peer reviewers agree on manuscript quality? JAMA. 1994;272:117–9.
4. Horrobin DF. The philosophical basis of peer review and the suppression of innovation. JAMA. 1990;263:1438–41.
5. Barrows HS. Simulated patients in medical teaching. Can Med Assoc J. 1968;98:674–6.
Feinstein AR, Clinical Epidemiology: The Architecture of Clinical Research. Philadelphia, PA: W. B. Saunders, 1985.
Fraenkel JR, Wallen NE. How to Design and Evaluate Research in Education. 4th ed. New York: McGraw—Hill Higher Education, 2000.
Fincher RM (ed). Guidebook for Clerkship Directors. Washington, DC: Association of American Medical Colleges, 2000.
Review Criteria for Research Manuscripts
Joint Task Force of Academic Medicine and the GEA-RIME Committee