Share this article on:

The Top-Cited Articles in Medical Education: A Bibliometric Analysis

Azer, Samy A. MD, PhD, MEd, MPH

doi: 10.1097/ACM.0000000000000780
Reviews
AM Rounds Blog Post

Purpose To identify and examine the characteristics of the 50 top-cited articles in medical education.

Method Two searches were conducted in the Web of Knowledge database in March 2014: a search of medical education journals in the category “Education, Scientific Discipline” (List A) and a keyword search across all journals (List B). Articles identified were reviewed for citation count, country of origin, article type, journal, authors, and publication year.

Results Both lists included 56 articles, not 50, because articles with the same absolute number of citations shared the same rank. The majority of List A articles were published in Academic Medicine (34; 60.7%) and Medical Education (16; 28.6%). In List B, 27 articles (48.2%) were published in medical education journals, 19 (33.9%) in general medicine and surgery journals, and 10 (17.9%) in higher education and educational psychology journals. Twenty-six articles were included in both lists, with different rankings. Reviews and articles constituted the majority of articles; there were only 8 research papers in List A and 13 in List B. Articles mainly originated from the United States, Canada, the Netherlands, and the United Kingdom. The majority were published from 1979 to 2007. There was no correlation between year and citation count.

Conclusions The finding that over half of List B articles were published in nonmedical education journals is consistent with medical education’s integrated nature and subspecialty breadth. Twenty of these articles were among their respective non-medical-education journals’ 50 top-cited papers, showing that medical education articles can compete with subject-based articles.

S.A. Azer is professor of medical education and chair, Curriculum Development and Research Unit, Medical Education Department, College of Medicine, King Saud University, Riyadh, Saudi Arabia.

Funding/Support: This work was funded by the College of Medicine Research Center, Deanship of Scientific Research, King Saud University, Riyadh, Saudi Arabia.

Other disclosures: None reported.

Ethical approval: Reported as not applicable.

Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A285.

An AM Rounds blog post on this article is available at academicmedicineblog.org.

Correspondence should be addressed to Samy A. Azer, Curriculum Development and Research Unit, Medical Education Department, College of Medicine, King Saud University, PO Box 2925, Riyadh 11461, Saudi Arabia; telephone: (966) 11-4699178; e-mail: azer2000@optusnet.com.au.

In the scientific community, number of citations is commonly used as a parameter in evaluating the impact of a researcher’s work or of a publication.1 Citation frequency is important not just for authors and journals but also for the universities and the nations where the work was completed.2 Although research funding is often highlighted and used as an indicator in quantifying the research activity of academic institutions,3 yearly citation ratings can be another parameter used in assessing university rankings globally.4 Citation ratings can also reflect the impact of a country in terms of research productivity and improving the quality of life of its citizens.5

The number of citations attracted by an article can be used in assessing that article’s relative significance in a particular field.6,7 Therefore, identifying articles that have exerted the most citation influence on a particular field could help scholars become familiar with “classic,” landmark works and highlight topics for further research in the area. Recent studies have identified the top-cited articles in microsurgery,8 urology,9 pediatric orthopedic surgery,10 orthopedic surgery,11 foot and ankle surgery,12 rehabilitation,13 “respiratory system,”14 complementary medicine,15 and nursing.16 However, to date, no studies have identified the top-cited articles in medical education. Therefore, the aim of this study was to identify the 50 most frequently cited articles in medical education published in journals with impact factors and to examine their characteristics. To make the study more comprehensive, two lists of top-cited medical education articles were constructed: one list of articles identified from medical education journals included in the Web of Knowledge category for science education and one list of articles identified by a keyword search across journals listed in the Web of Knowledge to capture medical education articles published in general medicine and specialty journals.

Back to Top | Article Outline

Method

Search strategy and study selection

Although Scopus and Google Scholar provide citation tracking, the Web of Knowledge was used to conduct this research because it covers cited publications back to the 1900s. Furthermore, it covers the Science Citation Index Expanded (1900–present) and the Social Sciences Citation Index (1900–present). The Web of Knowledge is regularly updated, enabling researchers to trace changes in numbers of citations over the years. In contrast, Google Scholar is difficult to search and the citation records include textbooks, non-peer-reviewed papers, and articles published in languages other than English. Scopus was not considered because the breadth of its years of coverage is not nearly as great as that of the Web of Knowledge.

Back to Top | Article Outline

Searching medical education journals

From March 1 to 12, 2014, the author and two research assistants used the search tools available within the Web of Knowledge to retrieve the top-cited articles published in medical education journals since 1900. A total of 34 journals were listed in the Web of Knowledge category “Education, Scientific Discipline”; among them, only 13 journals were related to medical education. These 13 journals were Academic Medicine; Advances in Health Sciences Education: Theory and Practice; Advances in Physiology Education; American Journal of Pharmaceutical Education; Biochemistry and Molecular Biology Education; BMC Medical Education; CBE–Life Sciences Education; Journal of Biological Education; Journal of Continuing Education in the Health Professions; Journal of Surgical Education; Medical Education; Medical Teacher; and Teaching and Learning in Medicine.

The articles published in each of the 13 journals were sorted using the “sorted by” option “Times cited—highest to lowest.” The results showed each journal’s articles organized in a descending order, with the articles cited most frequently at the top of the list. A copy of the results was printed out for further analysis. The findings from the 13 journal searches were arranged in a single list on one Excel spreadsheet (Microsoft Excel 2010, Microsoft Corp., Redmond, Washington) in descending order based on the number of citations attracted by each article.

The author and the two research assistants independently applied the inclusion and exclusion criteria (described below) to assess each article in the list. The judgments were based on the abstract of each article. The outcomes were discussed in a meeting; no paper was excluded from the list by the author or the two research assistants, and all articles matched the inclusion criteria. The combined list was reviewed again with regard to the number of citations allocated to each article. The articles with the most citations were identified and ranked from 1 to 50 to create the final list of top-cited articles in medical education journals (List A). If articles shared the same absolute number of citations, they were given the same rank.

Back to Top | Article Outline

Searching using keywords

From March 1 to 12, 2014, the author and two research assistants conducted a second search in Web of Knowledge using keywords to identify the top-cited medical education articles in any journal listed in the Web of Knowledge regardless of journal subject category. The aim of this search was to identify highly cited articles published since 1900 across all journals listed in the Web of Knowledge, not just those published in journals listed under the category “Education, Scientific Discipline.” The list created by this keyword search (List B) was compared with the list created by the category search described above (List A).

The keyword search used the following terms: problem-based learning, case-based learning, team-based learning, medical students, medical simulation, multiple-choice questions, extended matching questions, mini-CEX, OSCE, formative assessment, summative assessment, continuing assessment, online assessment, self-development, staff training, curriculum, curriculum outcomes, program evaluation, self-regulated learning, self-directed learning, professionalism, medical ethics, clinical skills, simulated patient, standardized patient, medical education, clinical teacher, facilitator, tutor, teaching basic science, learning anatomy, student feedback, feedback, role model, mentoring, clerkship, medical informatics, and e-learning. These keywords were identified using the taxonomy used in medical education journals and the proceedings of major medical education conferences such as the annual meetings of the Association of American Medical Colleges, the Association for Medical Education in Europe, the Canadian Conference on Medical Education, and the Association for the Study of Medical Education. Using Web of Knowledge tools, the results for each keyword were arranged using the “sorted by” option “Times cited—highest to lowest.” For each keyword, the most frequently cited articles were identified. A copy of the results for each keyword was printed out for further analysis. The outcomes for each keyword were placed in one Excel spreadsheet, where the articles were arranged in a single list from highest to lowest number of citations.

The author and the two research assistants independently applied the inclusion and exclusion criteria (described below) to assess each article in the list. The judgments were based on article abstracts. The outcomes of this process were discussed in a meeting; any differences were resolved through discussion after reading the full article. The articles with the most citations were ranked from 1 to 50 to create the final list of top-cited medical education articles across all journals (List B). If articles shared the same absolute number of citations, they were given the same rank.

Back to Top | Article Outline

Inclusion and exclusion criteria

Articles eligible for inclusion were those addressing areas of interest in medical education and to the medical education community and published in journals listed in the Web of Knowledge (i.e., journals with impact factors). Medical education was defined as a discipline covering the education of and the practice of skills by students enrolled in medical schools/colleges to become doctors; residency training; continuing medical education; and specialized postgraduate medical training (e.g., by organizations such as the Royal College of Physicians in the United Kingdom and the Royal Australasian College of Physicians in Australia and New Zealand).

Educational systems and assessment methods vary by specialty, governing body, and country. However, throughout the different stages of medical education, the discipline of medical education focuses on a number of subspecialties including curriculum design and implementation, assessment, faculty development and training, medical technology and medical informatics, medical education research, medical professionalism, teaching and learning methods, problem-based learning, and clinical teaching.17 The keywords used in our second search covered most subspecialties in medical education.

Therefore, the inclusion criteria for this study were articles published in journals listed in the Web of Knowledge and in the English language from 1900 to the date of the search covering (1) topics on medical education; (2) medical education issues related to medical students, interns, residents, or physicians; and/or (3) educational principles/theories from education or psychology that can be of value to the medical education community. The exclusion criteria were (1) articles in languages other than English or reviews including studies not published in English and (2) articles focused on education issues not of interest to the medical education community (e.g., articles on mathematics, engineering, or physics education, or other areas that do not target the heart of medical education).

Back to Top | Article Outline

Assessing the articles

The full text for each article included in List A or List B was obtained from the Saudi Digital Library (www.digitallibraryplus.com/). Following the methods of Lefaivre et al,11 each article was reviewed and the following information was recorded: (1) the first author’s name and affiliation; (2) organization and country of origin (when there was more than one author, the first author’s information was used); (3) number of citations; (4) year of publication; and (5) article type (review, article, educational guide, or research paper). For the purposes of this study, reviews were defined as articles reviewing recent progress in a particular topic, critically reviewing the current status of the literature and creating an understanding of the topic by discussing related literature, and/or identifying gaps in knowledge and highlighting future directions for research. Articles were defined as original reports with conclusions that represent a substantial advance in the understanding of important problems or issues. They provoke thought and ideas, and they aim to establish new directions. Educational guides were defined as articles written by a team of experts and aimed at providing teachers and educators with a resource on principles, current evidence, applications, and regulations. Educational guides are usually used as reference resources. Research papers were defined as systematic investigations into and the study of problems using valid and reliable methods in order to establish new findings and conclusions. Research papers may use qualitative, quantitative, or mixed methods.

Back to Top | Article Outline

Evaluating the journals

The journals that published the articles identified using keyword searches (List B) were evaluated. For these journals, the following key information was collected: (1) 2012 Journal Impact Factor; (2) language; (3) discipline; (4) number of citations for the article ranked no. 1 (i.e., the article with the highest number of citations among all articles published in the journal); and (5) number of citations of the article ranked no. 50 (i.e., the article ranked 50th among all articles published in the journal). Web of Knowledge citations numbers were used to identify the articles ranked no. 1 and no. 50 for each journal. The aim was to check whether the 50 top-cited articles identified by keywords in non-medical-education journals were among the 50 top-cited articles in each of the journals in which they were published.

Back to Top | Article Outline

Statistical analysis

Analysis was conducted using SPSS version 18.0 for Microsoft Windows (SPSS Inc., Chicago, Illinois), and results were reported as totals and percentages. The Pearson correlation coefficient (r) was calculated to determine whether the number of years since publication was correlated to the total number of citations.18 This relationship was examined to see whether the high numbers of citations obtained by these top-cited articles were related to their age rather than to the quality and the intrinsic value of their content.

Back to Top | Article Outline

Degree of agreement between assessors

The agreement between the three assessors was calculated using Cohen’s kappa for interrater correlation.19,20 The scores were in the range of 0.78 to 0.96 for articles in List B (articles identified via keyword search).

Back to Top | Article Outline

Results

Top-cited articles in the 13 medical education journals (List A)

Appendix 1 summarizes the top-cited articles published in the 13 medical education journals (List A). Articles are listed in descending order by rank from 1 to 50 based on the number of citations obtained by each article as of the day of the search.21–76 List A includes 56 articles, not 50, because some articles shared the same absolute number of citations. A denominator of 56 was used in the calculation of percentages. All articles were published in the English language.

The majority of these articles were reviews (n = 27; 48.2%) and articles (n = 19; 33.9%). Eight (14.3%) were research papers, and 2 (3.6%) were educational guides. The topics covered included problem-based learning (n = 9; 16.1%); clinical competency and assessment (n = 8; 14.3%); professionalism and professional competence/practice (n = 6; 10.7%); medical simulation and standardized patients (n = 4; 7.1%); faculty development (n = 4; 7.1%); and cross-cultural education, interprofessional education, and ethical environment (n = 4; 7.1 %). Table 1 summarizes other topics covered and topic distribution by article type.

Table 1

Table 1

These articles were published in five journals: Academic Medicine (n = 34; 60.7%), Medical Education (n = 16; 28.6%), Medical Teacher (n = 4; 7.1%), Journal of Continuing Education in the Health Professions (n = 1; 1.8%), and Advances in Health Sciences Education: Theory and Practice (n = 1; 1.8%).

The most frequently cited article was “Problem-based learning: A review of literature on its outcomes and implementation issues” by Albanese and Mitchell,21 which was published in Academic Medicine in 1993 and cited 861 times over 21 years. The article ranked 50th was “The New Mexico experiment: Educational innovation and institutional change” by Kaufman et al,76 which was published in Academic Medicine in 1989 and cited 141 times over 25 years. Most papers originated from academic institutions, with first authors from the United States (n = 31; 55.3%), Canada (n = 11; 19.6%), the United Kingdom (n = 6; 10.7%), the Netherlands (n = 5; 8.9%), Australia (n = 2; 3.6%), and Israel (n = 1; 1.8%).

All of the articles were published during a 28-year period (1979–2007) (see Table 2). The number of highly cited articles increased significantly after 1990, and the majority (n = 47; 83.9%) were published in the period from 1991 to 2007. No correlation was found between the year of publication and the number of citations obtained by an article (R2 = 0.054) (see Supplemental Digital Figure 1A, available at http://links.lww.com/ACADMED/A285).

Table 2

Table 2

Some authors contributed to more than one article in this list. Schmidt authored two of the most frequently cited articles33,46 and coauthored with Norman three others23,24,43 (see Appendix 1). These articles focused on problem-based learning (four articles) and cognitive perspectives on medical experts (one article). Norman was also the author55 or coauthor62 of two other articles on clinical reasoning and cognitive psychology in relation to professionalism, respectively. Irby authored two articles36,59 and coauthored a third.48 These articles were on teaching and learning in the ambulatory care setting, clinical teachers, and strategies for improving teaching practices, respectively. Hafferty authored an article on the hidden curriculum31 and coauthored another on the same topic.29 Van der Vleuten authored35 or coauthored44 two articles, both on assessment of professional competence. Harden was the first author of two articles.28,57 The first was a famous article on objective structured clinical examinations, and the second was on outcome-based education.

Back to Top | Article Outline

Top-cited articles identified by keyword search (List B)

Appendix 2 summarizes the top-cited medical education articles identified by keyword search across all journals (List B). Articles are listed in descending order by rank from 1 to 50 based on the number of citations received by each article as of the day of the search.21–39,41–47,77–106 List B includes 56 articles, not 50, because some articles shared the same absolute number of citations. A denominator of 56 was used in the calculation of percentages. All articles were published in the English language.

The majority of these articles were reviews (n = 30; 53.6%). The rest were articles (n = 13; 23.2%) or research papers (n = 13; 23.2%). There were no educational guides identified. The topics covered included problem-based learning (n = 11; 19.6%); clinical competency and assessment (n = 13; 23.2%); professionalism and professional competence (n = 4; 7.1%); medical simulation and standardized patients (n = 5; 8.9%); curriculum and educational innovation (n = 4; 7.1%); communication skills (n = 5; 8.9%); and medical experts and performance (n = 4; 7.1%). There were no articles on faculty development or error in diagnosis. Table 1 summarizes other topics covered and topic distribution by article type.

These articles were published in 21 journals. Twenty-seven (48.2%) of the articles were published in 5 medical education journals (2012 Journal Impact Factor; no. of articles): Academic Medicine (3.292; n = 17), Medical Education (3.546; n = 7), Medical Teacher (1.824; n = 1), Journal of Continuing Education in the Health Professions (1.321; n = 1), and Advances in Health Sciences Education: Theory and Practice (2.061; n = 1).

The remaining articles were published in two groups of journals. Nineteen (33.9%) were published in the group of 10 general medicine and surgery journals (2012 Journal Impact Factor; no. of articles): JAMA–Journal of the American Medical Association (29.978; n = 7), CMAJ–Canadian Medical Association Journal (6.465; n = 1), Annals of Internal Medicine (13.976; n = 2), Archives of Internal Medicine (10.579; n = 1), BMJ–British Medical Journal (17.215; n = 1), Journal of the American College of Surgeons (4.500; n = 1), NEJM–The New England Journal of Medicine (51.658; n = 3), The Lancet (39.060; n = 1), BJA–British Journal of Anaesthesia (4.237; n = 1), and Annals of Surgery (6.329; n = 1).

The other 10 articles (17.9%) were published in the second group of journals, which included 4 higher education and 3 educational psychiatry journals (2012 Journal Impact Factor; no. of articles): Review of Educational Research (4.229; n = 2), Journal of the Learning Sciences (3.036; n = 1), Studies in Higher Education (1.036; n = 1), Learning and Instruction (3.337; n = 1), Educational Psychologist (3.289; n = 3), Educational Psychology Review (2.154; n = 1), and Assessment (2.430; n = 1).

The most frequently cited article was “Evidence-based medicine: A new approach to teaching the practice of medicine” by the Evidence-Based Medicine Working Group, published in JAMA–Journal of the American Medical Association in 1992 and cited 1,278 times over 22 years.77 Two articles shared the rank of 50th: “Do clinical clerks suffer ethical erosion? Students’ perceptions of their ethical environment and personal development” by Feudtner et al,47 which was published in Academic Medicine in 1994 and cited 213 times over 20 years, and “The mini-CEX: A method for assessing clinical skills” by Norcini et al,106 which was published in Annals of Internal Medicine in 2003 and cited 213 times over 11 years. Most articles originated from academic institutions, with first authors from the United States (n = 35; 62.5%), Canada (n = 8; 14.3%), the Netherlands (n = 6; 10.7%), the United Kingdom (n = 4; 7.1%), Australia (n = 1; 1.8%), Belgium (n = 1; 1.8%), and New Zealand (n = 1; 1.8%).

The articles were published within a 28-year period (1979–2007) (see Table 2). The number of highly cited articles increased significantly after 1990, and the majority (n = 50; 89.3%) were published in the period from 1991 to 2007 (Table 2). There was no correlation between the year of publication and the number of citations obtained by an article (R2 = 0.0033) (see Supplemental Digital Figure 1B at http://links.lww.com/ACADMED/A285).

Further assessment of the articles in List B revealed that 20 of the top-cited medical education articles published in non-medical-education journals were also among their respective journals’ 50 top-cited articles: JAMA–Journal of the American Medical Association,77,84,85,92,97CMAJ–Canadian Medical Association Journal,78Annals of Internal Medicine,80BMJ–British Medical Journal,91Journal of the American College of Surgeons,93BJA–British Journal of Anaesthesia,100Educational Psychologist,81,88,89Review of Educational Research,82,87Assessment,83Journal of the Learning Sciences,86Educational Psychology Review,94Studies in Higher Education,95 and Learning and Instruction.101

Again in List B, the two articles Schmidt authored33,46 and the three articles Schmidt coauthored with Norman23,24,43 were among the top-cited articles, as were the two papers on the hidden curriculum authored or coauthored by Hafferty.29,31 List B included three articles by Van Der Vleuten on assessment,35,44,99 two of which were also included in List A.35,44 Epstein authored two articles85,96 and coauthored two articles79,97 on assessing professional competence and mindful practice of assessment. Also, Papadakis coauthored two articles71,104 on unprofessional behavior and disciplinary action by medical boards (see Appendix 2).

Back to Top | Article Outline

Articles included in both lists

Comparison of the top-cited articles included in the two lists—articles identified by searching medical education journals (category search; List A) or by searching using keywords (List B)—revealed that there were 26 articles identified by both searches.21–39,41–47

Back to Top | Article Outline

Discussion

The aim of this study was to identify the 50 most frequently cited articles in medical education and to examine their characteristics in order to gain insight into the history and main developments of the medical education field. To ensure that articles published in journals other than journals that specialized in medical education were identified, two searches were conducted in the Web of Knowledge. The first search was a category search covering 13 medical education journals (List A), and the second search used keywords related to major medical education topics covered under medical education to search across all journals listed in the Web of Knowledge (List B). In both searches, most of the top-cited articles identified originated from the United States, Canada, the United Kingdom, or the Netherlands, and the top-cited articles were published in the English language. Similar findings have been reported in other fields.16,107

The findings from this study show that there were 26 articles included in both lists, but the rankings of these articles differed in List A (Appendix 1) and List B (Appendix 2). The keyword search revealed that there were top-cited papers on medical education published in journals other than those that specialized in medical education: 29 (51.8%) of the 56 articles included in List B were not published in medical education journals. These non-medical-education journals had 2012 Journal Impact Factors ranging from 1.036 (Studies in Higher Education) to 51.658 (The New England Journal of Medicine), and they covered a range of disciplines including general medicine, surgery, higher education, and educational psychology. The finding that more than half of the articles in List B were published in non-medical-education journals is consistent with the integrated nature of medical education and the breadth of its subspecialties. It also shows the interest of general medicine journals and some surgery journals in publishing articles on topics related to medical education.

It is of interest to note that articles published in general medicine journals attracted more citations than those published in medical education journals. However, the range of numbers of citations for medical education articles was considerably lower than the range of numbers of citations obtained by classic articles in other disciplines such as surgery, medicine, and pediatrics.9,11,14

The finding that 20 of the top-cited medical education papers published in general medicine, educational psychology, and higher education journals were also among the 50 top-cited papers in their respective journals shows the ability of these articles to compete with subject-based articles published in these specialized journals.

The topics covered by the articles in List A and List B were nearly the same, although the number of articles in each topic varied. Interestingly, in both lists, the majority of the articles were not research papers: Only 8 articles (14.3%) in List A and 13 articles (23.2%) in List B were research papers (see Table 1). Four research papers were included in both lists.37,41,45,47 Most of the research papers were qualitative studies (n = 5 in List A, n = 8 in List B); the others were quantitative studies (n = 3 in List A, n = 5 in List B). The methods used in qualitative studies in both lists included face-to-face interviews, questionnaires, observations, and anonymous mailed surveys with follow-up letters and/or telephone surveys. Only 3 research papers used a randomized controlled study design,84,90,93 and 2 were case–control studies.71,104 The small number of randomized controlled and case–control studies in medical education research is related to the difficulty in designing such studies in an educational program.108

Two other studies are of interest to this work.109,110 Sampson et al109 examined the characteristics of medical education studies published in five general and internal medicine (GIM) journals and five medical education journals. They reported that medical education journals had a concentration of studies targeting medical students, whereas GIM journals had a concentration of studies targeting residents. Lee et al110 aimed at assaying the nature and context of growth in publications in the medical education field from 1960 to 2010 by searching MEDLINE for articles on medical education. They found that the annual publications in medical education increased from 279 in 1960 to 3,760 in 2010. They concluded that the increases in the number of medical education publications and in the number of journals publishing medical education articles suggest growth in the field.

The present work and the study by Sampson et al109 each found similarities in the themes of publications in both groups of journals, while the distributions for each topic or category were different. The findings of Lee et al110 are in agreement with the present study. The numbers of citations obtained by the top-cited articles in List A and List B and the finding that articles on medical education were among the top-cited articles published in general medicine journals are indications of the increasing number of publications in medical education and the development of new investigations in the field. Lee and colleagues’110 findings that authors from the United States, Canada, and the Netherlands dominated the authorship are also consistent with this study’s findings.

This study provides insight into knowledge generation processes that are vital in medical education practices and research work. It also honors the research leaders in the field by identifying their work that has been frequently cited by other researchers over three decades.

Back to Top | Article Outline

Limitations and strengths

There are a few limitations of this study. First, the search methods used were based on journals with impact factors and articles published in the English language. This may have contributed to the high number of articles from Western countries, especially the United States, Canada, the Netherlands, and the United Kingdom. Therefore, influential articles in languages other than English that may have global impact were not included. Second, factors that may affect the citation rates of an article were not carefully addressed. For example, authors’ self-citations were not excluded from the total number of citations, and the absolute number of citations was used to determine rank.111 Third, the searches were conducted using only one database, the Web of Knowledge, and there is the possibility that the database’s filters are not sensitive enough. For example, the article by the Medical School Objectives Writing Group103 published in Academic Medicine was not identified when the search was conducted for articles published in that journal, but it was identified in the keyword search; therefore, it was included in List B but not List A. The opposite was the case for the paper by Croskerry,40 which was included in List A but not List B. Fourth, focusing on the 50 top-cited articles represents an arbitrary number. Finally, the articles included in this study represent the search outcomes at the time the searches were conducted.

One of the strengths of this study is the use of two search methods to identify the top-cited articles. The keyword search enabled the identification of top-cited papers in medical education that were published in non-medical-education journals.

Back to Top | Article Outline

Conclusions

This bibliometric study identified the most frequently cited medical education papers published in medical education journals (List A) and across all journals listed in the Web of Knowledge (List B). The results show the impact that the articles have had on the scientific community and the field of medical education. The “classic” articles identified in this study are landmark papers that have contributed greatly to the field. Evidence of influential scholarship such as what is shown in this study could be considered as one criterion for awards in medical education scholarship such as the Karolinska Institutet awards.

Acknowledgments: The author would like to thank Dr. Sarah Azer and Diana Azer for their assistance and reviewing the manuscript. He also thanks Ms. Laniefel Mendoza (Peng) and Ms. Mary Hadlocon for their secretarial help. The author is grateful to the reviewers and the editor-in-chief of Academic Medicine for their constructive feedback that helped him improve the manuscript. The detailed report from the editor and the editorial staff member was very helpful. The author would like to thank them for their time and effort.

Back to Top | Article Outline

References

1. Moed HF. New developments in the use of citation analysis in research evaluation. Arch Immunol Ther Exp (Warsz). 2009;57:13–18
2. Rosmarakis ES, Vergidis PI, Soteriades ES, Paraschakis K, Papastamataki PA, Falagas ME. Estimates of global production in cardiovascular diseases research. Int J Cardiol. 2005;100:443–449
3. Ellwein LB, Khachab M, Waldman RH. Assessing research productivity: Evaluating journal publication across academic departments. Acad Med. 1989;64:319–325
4. van Raan AF, van Leeuwen TN, Visser MS. Severe language effect in university rankings: Particularly Germany and France are wronged in citation-based rankings. Scientometrics. 2011;88:495–498
5. Doi H, Heeren A, Maurage P. Scientific activity is a better predictor of Nobel award chances than dietary habits and economic factors. PLoS One. 2014;9:e92612 http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0092612. Accessed April 21, 2015
6. Kanter SL. Understanding the journal’s impact. Acad Med. 2009;84:1169–1170
7. Iyengar R, Wang Y, Chow J, Charney DS. An integrated approach to evaluate faculty members’ research performance. Acad Med. 2009;84:1610–1616
8. Joyce CW, Carroll SM. Microsurgery: The top 50 classic papers in plastic surgery: A citation analysis. Arch Plast Surg. 2014;41:153–157
9. Heldwein FL, Rhoden EL, Morgentaler A. Classics of urology: A half century history of the most frequently cited articles (1955–2009). Urology. 2010;75:1261–1268
10. Baldwin KD, Kovatch K, Namdari S, Sankar W, Flynn JM, Dormans JP. The 50 most cited articles in pediatric orthopedic surgery. J Pediatr Orthop B. 2012;21:463–468
11. Lefaivre KA, Shadgan B, O’Brien PJ. 100 most cited articles in orthopaedic surgery. Clin Orthop Relat Res. 2011;469:1487–1497
12. Bayley M, Brooks F, Tong A, Hariharan K. The 100 most cited papers in foot and ankle surgery. Foot (Edinb). 2014;24:11–16
13. Shadgan B, Roig M, Hajghanbari B, Reid WD. Top-cited articles in rehabilitation. Arch Phys Med Rehabil. 2010;91:806–815
14. Tam WW, Wong EL, Wong FC, Hui DS. Citation classics: Top 50 cited articles in “respiratory system.” Respirology. 2013;18:71–81
15. Tam WWS, Wong ELY, Wong FCY, Cheung AWL. Citation classics in the integrative and complementary medicine literature: 50 frequently cited articles. Eur J Integr Med. 2012;4:e77–e83
16. Wong EL, Tam WW, Wong FC, Cheung AW. Citation classics in nursing journals: The top 50 most frequently cited articles from 1956 to 2011. Nurs Res. 2013;62:344–351
17. The MedEdWorld Glossary. http://www.mededworld.org/Glossary.aspx. Accessed July 20, 2014
18. Bryman A Social Research Methods. 20084th ed New York, NY Oxford University Press
19. Wongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. BMC Med Res Methodol. 2013;13:61 http://www.biomedcentral.com/1471–2288/13/61. Accessed April 21, 2015
20. Haley SM, Osberg JS. Kappa coefficient calculation using multiple ratings per subject: A special communication. Phys Ther. 1989;69:970–974
21. Albanese MA, Mitchell S. Problem-based learning: A review of literature on its outcomes and implementation issues. Acad Med. 1993;68:52–81
22. Issenberg SB, McGaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: A BEME systematic review. Med Teach. 2005;27:10–28
23. Norman GR, Schmidt HG. The psychological basis of problem-based learning: A review of the evidence. Acad Med. 1992;67:557–565
24. Schmidt HG, Norman GR, Boshuizen HP. A cognitive perspective on medical expertise: Theory and implication. Acad Med. 1990;65:611–621
25. Vernon DT, Blake RL. Does problem-based learning work? A meta-analysis of evaluative research. Acad Med. 1993;68:550–563
26. Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;79(10 suppl):S70–S81
27. Graham ID, Logan J, Harrison MB, et al. Lost in knowledge translation: Time for a map? J Contin Educ Health Prof. 2006;26:13–24
28. Harden RM, Gleeson FA. Assessment of clinical competence using an objective structured clinical examination (OSCE). Med Educ. 1979;13:41–54
29. Hafferty FW, Franks R. The hidden curriculum, ethics teaching, and the structure of medical education. Acad Med. 1994;69:861–871
30. Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 suppl):S63–S67
31. Hafferty FW. Beyond curriculum reform: Confronting medicine’s hidden curriculum. Acad Med. 1998;73:403–407
32. Barrows HS. A taxonomy of problem-based learning methods. Med Educ. 1986;20:481–486
33. Schmidt HG. Problem-based learning: Rationale and description. Med Educ. 1983;17:11–16
34. Colliver JA. Effectiveness of problem-based learning curricula: Research and theory. Acad Med. 2000;75:259–266
35. Van Der Vleuten CP. The assessment of professional competence: Developments, research and practical implications. Adv Health Sci Educ Theory Pract. 1996;1:41–67
36. Irby DM. Teaching and learning in ambulatory care settings: A thematic review of the literature. Acad Med. 1995;70:898–931
37. Regehr G, MacRae H, Reznick RK, Szalay D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med. 1998;73:993–997
38. Barrows HS. An overview of the uses of standardized patients for teaching and evaluating clinical skills. Acad Med. 1993;68:443–451
39. Ruiz JG, Mintzer MJ, Leipzig RM. The impact of E-learning in medical education. Acad Med. 2006;81:207–212
40. Croskerry P. The importance of cognitive errors in diagnosis and strategies to minimize them. Acad Med. 2003;78:775–780
41. Sutcliffe KM, Lewton E, Rosenthal MM. Communication failures: An insidious contributor to medical mishaps. Acad Med. 2004;79:186–194
42. Makoul G. Essential elements of communication in medical encounters: The Kalamazoo consensus statement. Acad Med. 2001;76:390–393
43. Norman GR, Schmidt HG. Effectiveness of problem-based learning curricula: Theory, practice and paper darts. Med Educ. 2000;34:721–728
44. van der Vleuten CP, Schuwirth LW. Assessing professional competence: From methods to programmes. Med Educ. 2005;39:309–317
45. Newble DI, Jaeger K. The effect of assessments and examinations on the learning of medical students. Med Educ. 1983;17:165–171
46. Schmidt HG. Foundations of problem-based learning: Some explanatory notes. Med Educ. 1993;27:422–432
47. Feudtner C, Christakis DA, Christakis NA. Do clinical clerks suffer ethical erosion? Students’ perceptions of their ethical environment and personal development. Acad Med. 1994;69:670–679
48. Wilkerson L, Irby DM. Strategies for improving teaching practices: A comprehensive approach to faculty development. Acad Med. 1998;73:387–396
49. Downing SM. Validity: On meaningful interpretation of assessment data. Med Educ. 2003;37:830–837
50. Dyrbye LN, Thomas MR, Shanafelt TD. Systematic review of depression, anxiety, and other indicators of psychological distress among U.S. and Canadian medical students. Acad Med. 2006;81:354–373
51. Gordon MJ. A review of the validity and accuracy of self-assessments in health professions training. Acad Med. 1991;66:762–769
52. Steinert Y, Mann K, Centeno A, et al. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME guide no. 8. Med Teach. 2006;28:497–526
53. Ziv A, Wolpe PR, Small SD, Glick S. Simulation-based medical education: An ethical imperative. Acad Med. 2003;78:783–788
54. Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: From Flexner to competencies. Acad Med. 2002;77:361–367
55. Norman G. Research in clinical reasoning: Past history and current trends. Med Educ. 2005;39:418–427
56. Newble DI, Entwistle NJ. Learning styles and approaches: Implications for medical education. Med Educ. 1986;20:162–175
57. Harden RM, Crosby JR, Davis MH, Friedman M. AMEE guide no. 14: Outcome-based education: Part 5—From competency to meta-competency: A model for the specification of learning outcomes. Med Teach. 1999;21:546–552
58. Bland CJ, Meurer LN, Maldonado G. Determinants of primary care specialty choice: A non-statistical meta-analysis of the literature. Acad Med. 1995;70:620–641
59. Irby DM. What clinical teachers in medicine need to know. Acad Med. 1994;69:333–342
60. Swick HM. Toward a normative definition of medical professionalism. Acad Med. 2000;75:612–616
61. Arnold L. Assessing professional behavior: Yesterday, today, and tomorrow. Acad Med. 2002;77:502–515
62. Regehr G, Norman GR. Issues in cognitive psychology: Implications for professional education. Acad Med. 1996;71:988–1001
63. Massaro TA. Introducing physician order entry at a major academic medical center: II. Impact on medical education. Acad Med. 1993;68:25–30
64. Kneebone R. Simulation in surgical training: Educational issues and practical implications. Med Educ. 2003;37:267–277
65. Hammick M, Freeth D, Koppel I, Reeves S, Barr H. A best evidence systematic review of interprofessional education: BEME guide no. 9. Med Teach. 2007;29:735–751
66. Eva KW. What every teacher needs to know about clinical reasoning. Med Educ. 2005;39:98–106
67. Samkoff JS, Jacques CH. A review of studies concerning effects of sleep deprivation and fatigue on residents’ performance. Acad Med. 1991;66:687–693
68. Kilminster SM, Jolly BC. Effective supervision in clinical practice settings: A literature review. Med Educ. 2000;34:827–840
69. Hall P, Weaver L. Interdisciplinary education and teamwork: A long and winding road. Med Educ. 2001;35:867–875
70. Walton HJ, Matthews MB. Essentials of problem-based learning. Med Educ. 1989;23:542–558
71. Papadakis MA, Hodgson CS, Teherani A, Kohatsu ND. Unprofessional behavior in medical school is associated with subsequent disciplinary action by a state medical board. Acad Med. 2004;79:244–249
72. Faulkner H, Regehr G, Martin J, Reznick R. Validation of an objective structured assessment of technical skill for surgical residents. Acad Med. 1996;71:1363–1365
73. Hojat M, Mangione S, Nasca TJ, et al. An empirical study of decline in empathy in medical school. Med Educ. 2004;38:934–941
74. Betancourt JR. Cross-cultural medical education: Conceptual approaches and frameworks for evaluation. Acad Med. 2003;78:560–569
75. Palepu A, Friedman RH, Barnett RC, et al. Junior faculty members’ mentoring relationships and their professional development in U.S. medical schools. Acad Med. 1998;73:318–323
76. Kaufman A, Mennin S, Waterman R, et al. The New Mexico experiment: Educational innovation and institutional change. Acad Med. 1989;64:285–294
77. Evidence-Based Medicine Working Group. . Evidence-based medicine. A new approach to teaching the practice of medicine. JAMA. 1992;268:2420–2425
78. Stewart MA. Effective physician–patient communication and health outcomes: A review. CMAJ. 1995;152:1423–1433
79. Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA. 2002;287:226–235
80. ABIM Foundation. . American Board of Internal Medicine; ACP–ASIM Foundation. American College of Physicians–American Society of Internal Medicine; European Federation of Internal Medicine. Medical professionalism in the new millennium: A physician charter. Ann Intern Med. 2002;136:243–246
81. Kirschner PA, Sweller J, Clark RE. Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educ Psychol. 2006;41:75–86
82. Butler DL, Winne PH. Feedback and self-regulated learning—A theoretical synthesis. Rev Educ Res. 1995;65:245–281
83. Baer RA, Smith GT, Hopkins J, Krietemeyer J, Toney L. Using self-report assessment methods to explore facets of mindfulness. Assessment. 2006;13:27–45
84. Peabody JW, Luck J, Glassman P, Dresselhaus TR, Lee M. Comparison of vignettes, standardized patients, and chart abstraction: A prospective validation study of 3 methods for measuring quality. JAMA. 2000;283:1715–1722
85. Epstein RM. Mindful practice. JAMA. 1999;282:833–839
86. Anderson JR, Corbett AT, Koedinger KR, Pelletier R. Cognitive tutor: Lessons learned. J Learn Sci. 1995;4:167–207
87. Hattie J, Timperley H. The power of feedback. Rev Educ Res. 2007;77:81–112
88. Deci EL, Vallerand RJ, Pelletier LG, Ryan RM. Motivation and education—the self-determination perspective. Educ Psychol. 1991;26:325–346
89. Schunk DH. Self-efficacy and academic motivation. Educ Psychol. 1991;26:207–231
90. Roter DL, Hall JA, Kern DE, Barker LR, Cole KA, Roca RP. Improving physicians’ interviewing skills and reducing patients’ emotional distress. A randomized clinical trial. Arch Intern Med. 1995;155:1877–1884
91. Simpson M, Buckman R, Stewart M, et al. Doctor–patient communication: The Toronto consensus statement. BMJ. 1991;303:1385–1387
92. Issenberg SB, McGaghie WC, Hart IR, et al. Simulation technology for health care professional skills training and assessment. JAMA. 1999;282:861–866
93. Scott DJ, Bergen PC, Rege RV, et al. Laparoscopic training on bench models: Better and more cost effective than operating room experience? J Am Coll Surg. 2000;191:272–283
94. Hmelo-Silver CE. Problem-based learning: What and how do students learn? Educ Psychol Rev. 2004;16:235–266
95. Nicol DJ, Macfarlane-Dick D. Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Stud Higher Educ. 2006;31:199–218
96. Epstein RM. Assessment in medical education. N Engl J Med. 2007;356:387–396
97. Novack DH, Suchman AL, Clark W, Epstein RM, Najberg E, Kaplan C. Calibrating the physician. Personal awareness and effective patient care. Working Group on Promoting Physician Personal Awareness, American Academy on Physician and Patient. JAMA. 1997;278:502–509
98. Gallagher AG, Ritter EM, Champion H, et al. Virtual reality simulation for the operating room: Proficiency-based training as a paradigm shift in surgical skills training. Ann Surg. 2005;241:364–372
    99. Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet. 2001;357:945–949
    100. Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Anaesthetists’ Non-Technical Skills (ANTS): Evaluation of a behavioural marker system. Br J Anaesth. 2003;90:580–588
    101. Dochy F, Segers M, Van den Bossche P, Gijbels D. Effects of problem-based learning: A meta-analysis. Learn Instr. 2003;13:533–568
    102. Wetzel MS, Eisenberg DM, Kaptchuk TJ. Courses involving complementary and alternative medicine at US medical schools. JAMA. 1998;280:784–787
      103. Medical School Objectives Writing Group. . Learning objectives for medical student education—guidelines for medical schools: Report I of the Medical School Objectives Project. Acad Med. 1999;74:13–18
      104. Papadakis MA, Teherani A, Banach MA, et al. Disciplinary action by medical boards and prior behavior in medical school. N Engl J Med. 2005;353:2673–2682
      105. Cooke M, Irby DM, Sullivan W, Ludmerer KM. American medical education 100 years after the Flexner report. N Engl J Med. 2006;355:1339–1344
      106. Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX: A method for assessing clinical skills. Ann Intern Med. 2003;138:476–481
      107. Gehanno JF, Takahashi K, Darmoni S, Weber J. Citation classics in occupational medicine journals. Scand J Work Environ Health. 2007;33:245–251
      108. Eva KW, Lingard L. What’s next? A guiding question for educators engaged in educational research. Med Educ. 2008;42:752–754
      109. Sampson M, Horsley T, Doja A. A bibliometric analysis of evaluative medical education studies: Characteristics and indexing accuracy. Acad Med. 2013;88:421–427
      110. Lee K, Whelan JS, Tannery NH, Kanter SL, Peters AS. 50 years of publication in the field of medical education. Med Teach. 2013;35:591–598
      111. Kulkarni AV, Aziz B, Shams I, Busse JW. Author self-citation in the general medicine literature. PLoS One. 2011;6:e20885 http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0020885. Accessed April 21, 2015
      Appendix 1

      Appendix 1

      Appendix 2

      Appendix 2

      Supplemental Digital Content

      Back to Top | Article Outline
      © 2015 by the Association of American Medical Colleges