Evidence-based medicine (EBM) uses the best evidence to support making health-related decisions with patients. EBM has 5 steps: asking a well-structured question, searching and retrieving evidence, critically appraising the evidence, applying the evidence while integrating patient values and preferences, and evaluating the process.1 Learning how to practice EBM is a critical component of health professional education.1,2 Effective application of this process has been linked to reduced medical error, promotion of individualized care, and increased application of best practices.3 Despite these benefits, students struggle with EBM skills.2 There is a need to improve current educational approaches to EBM training in undergraduate medical education (UME) programs.3
Optimal delivery of EBM training is not well defined. Key publications suggest that effective EBM teaching in UME incorporates 2 core principles: use of interactive and clinically integrated learning activities and inclusion of whole-task learning exercises (exercises using all the steps involved in a process to avoid fragmentation).2–4 Ideally, EBM content should be assessed using “authentic” assessments, whereby learners demonstrate higher-order thinking processes such as critical thinking and problem solving. The “authenticity” of an assignment comes from students’ perceptions of instructional effectiveness, alignment with professional roles, and meaningful achievement—rather than a focus on demonstrating the acquisition of knowledge or skills in various recall formats.5
Wikipedia is the most-read medical information platform on the Internet with over 32,000 medical articles viewed more than 10 million times daily.6 Wikipedia medical articles are widely accessed by health care trainees and practitioners across the continuum of their careers.7 Unlike other medical information resources, Wikipedia is compiled and maintained by volunteer editors. While contributions from unknown entities have raised concerns, comparisons of Wikipedia articles with other sources (textbooks, databases, and practice guidelines) have shown positive results, suggesting that the articles—depending on the subject—demonstrate reasonable accuracy.8 The quality of medical articles on Wikipedia is monitored by WikiProject Medicine (WP:MED), a large group of volunteer editors curating medical articles on Wikipedia. There is a need for ongoing revision of Wikipedia articles. As of December 2018 on English Wikipedia, over 150 medical articles were tagged as needing “expert attention,” and 92% were rated B class, which means they require improvements. Contributing to this improvement process by using Wikipedia editing as a framework to train learners satisfies the previously identified core principles of effective EBM teaching.
Several initiatives in higher education feature activities through which learners edit Wikipedia, but there are few in medicine. One example is the final-year elective offered at the University of California, San Francisco (UCSF) School of Medicine in which students self-select and edit a Wikipedia article.9 This pioneering elective informed our plan to structure EBM training for first-year students using Wikipedia. During the design, we communicated with UCSF instructors to discuss any lessons they had learned from offering the elective. Our course differs from that of USCF’s in key ways: it welcomes new medical students into a community of practice, uses a structured EBM task-based approach in the editing process, is generalizable to other faculties seeking a framework for teaching EBM early in UME, and facilitates a major contribution to Wikipedia by a substantially larger student group. In this report, we describe the implementation, evaluation, and ongoing improvement of a mandatory Wikipedia editing assignment as an authentic task for teaching EBM to first-year medical students.
MEDS 112 is a first-year undergraduate course at Queen’s University School of Medicine that covers content in clinical research design and the components of EBM. Although the course is well received by students, some have requested assessment tools that transcend recall-based examinations and capture critical thinking. In an effort to work toward this goal, in 2016, we joined an institution-wide endeavor aimed at improving course assessments to align with critical thinking and problem-solving outcomes. Designing and implementing the Wikipedia project was our contribution to this institutional assessment overhaul, and our local institutional research ethics board approved the project.
We developed an EBM assignment with the following objectives:
- Evaluate Wikipedia as a platform for sharing medical information;
- Search for and locate appropriate biomedical information;
- Enhance Wikipedia medical content using EBM skills; and
- Analyze the claims of a Wikipedia article on complementary and alternative medicine (CAM).
Collaborating with Wikipedia insiders was a priority to ensure success. One of us (H.M.) connected with WP:MED in August 2017 and recruited an established Wikipedia medical editor (J.D.) onto the teaching team. In the same month, we registered the course with WikiEdu, a nonprofit organization supporting instructors using Wikipedia articles for education. Students were oriented to Wikipedia in September 2017 by completing WikiEdu’s online training modules for new editors, and their editing activity was tracked through an interactive online dashboard. The modules provided guidance for sourcing medical content and an introduction to the mechanics of editing Wikipedia.
During the fall of 2017, students worked in small groups of 6 to 8 to complete this longitudinal project. The groups selected a Wikipedia article flagged by WP:MED as needing improvement and then worked incrementally on their article through a series of linked assignments over the fall term. We developed 4 specific assessments for the project, alternating between group-based and individual work: (1) a group-written critique of an existing medicine-related article on Wikipedia, along with a plan to improve it (September), (2) an individual literature search to support planned changes (October), (3) a group peer review of another group’s planned enhancements (November), and (4) an individual critique of a second article (December). For this individual culminating assignment, students were given 1 of 10 CAM articles available on Wikipedia. Assignment questions required them to demonstrate critical thinking in analyzing the claims and the quality of the supporting evidence; in explaining why they would or would not recommend this therapy; and, finally, in describing how they might respond to patient questions about CAM.
Upon finishing the course in December, all students completed an evaluation form asking about their perceptions of the project’s strengths and weaknesses. We also inquired about the struggles and successes they encountered while completing their project (see Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/A774). Two of us (M.W. and L.A.M.) deidentified and reviewed the feedback data in aggregate, seeking themes using the Five-Dimensional Framework for Authentic Assessment.5 We assigned students’ comments to 1 of the 5 dimensions relating to (1) the task, (2) the physical/virtual context, (3) the social context, (4) the result, and (5) the criteria for evaluation. We also determined if students’ comments identified barriers or facilitators to completing the project.
To capture the effectiveness of the CAM assignment in evaluating critical thinking, we invited students to have their course assignments scored by trained researchers, in addition to the faculty assessment that formed part of their course grade. Volunteered assignments received 2 sets of scores: one from faculty and one from the researchers. The researchers used the critical thinking Valid Assessment of Learning in Undergraduate Education (VALUE) rubric, considered a gold standard.10 The VALUE rubric has 5 performance levels and evaluates student performance in critical thinking domains, specifically (1) explanation of issues, (2) evidence, (3) context and assumptions, (4) position, and (5) conclusions and outcomes. Seventeen of the 101 students consented to have their assignments released and assessed again using the VALUE rubric. We compared the scores from the faculty facilitators with those of the researchers, using a Pearson correlation analysis.
During the fall of 2017, 101 students made over 1,000 unique edits, adding over 10,000 words to 16 articles. All article improvements included secondary source citations that met Wikipedia’s guideline for reliable sources. Our Wikipedia editor (J.D.) and volunteers from WP:MED moderated the edits online and provided the students with feedback on the article’s discussion page.
A sample of thematically organized student responses to the project evaluation survey is presented in Table 1. Based on feedback data, students enjoyed applying the critical appraisal skills taught within the broader scope of the course (task); they liked improving a highly accessed public resource (result); they reported positive collaboration within their teams (social context); and they enjoyed learning about the process of editing a Wikipedia medical article (task). Barriers to completing the project included lack of clarity regarding assignment expectations (task), frustration with Wikipedia coding (task), difficulty engaging with the Wikipedia editors/community (social context), distrust of Wikipedia editors as content experts (social context), and perceived mismatch of efforts dedicated to the assignment and the resulting change/effect on the Wikipedia medical article (result).
We noted moderate to high correlation between the grade for the 17 consenting individual student assignments and scores on 4 of the 5 externally scored VALUE rubric dimensions (see Table 2). This relatively high correlation supports the concurrent validity and statistical reliability for the culminating assessment of students’ critical thinking. The lowest correlation was between the course assignment mark and the “explanation of issues” dimension, suggesting that this is a lesser focus of the CAM assignment. The small sample size and voluntary nature of the duplicate marking (selection bias) are limitations which may overestimate these correlations.
The semester-long project demonstrates the feasibility of creating authentic assessment through Wikipedia editing, which entails the application of the core principles of high-quality EBM teaching. Initial results confirm the alignment of assessment structures in capturing elements of critical thinking and moving beyond fact-based recall. Structured course feedback highlights aspects students appreciated in the course, as well as barriers they encountered in completing the longitudinal Wikipedia project as part of their first-year critical appraisal course. Their comments highlighted a few key problems, which we began to address in the second (fall 2018) iteration of the course.
One problem identified by students and course faculty alike was the duplication of effort required when written assignments were separate from the online editing process. The assessment structure was redesigned in the second iteration (fall 2018) to evaluate the changes made to articles during the editing process online and in real time rather than requiring students to provide written summaries. Another issue was the hesitancy of first-year students to make edits even after carefully reviewing the literature. Faculty at UCSF did not observe this in the final-year students who elected to take the Wikipedia course,9 so this hesitation might have been a reflection of self-confidence and experience understanding and evaluating medical content. We have addressed this challenge by recruiting faculty tutors who select articles in their own area of expertise. These faculty have directly supported the student editing process, which we believe has increased student confidence and engagement with the medical content. Student reluctance to interact with the platform and the online community is expected to decrease as institutional familiarity increases. Further, while students enjoyed applying critical appraisal skills, some expressed frustration at a perceived mismatch in their efforts and the resulting effect on their Wikipedia article. We have worked to alleviate these concerns in the second iteration of the course as faculty facilitators guided students in managing expectations. One final barrier important for those wishing to implement this assignment structure at their own institutions is the time required for both students and faculty to learn how Wikipedia works and how to edit. Learning how to edit Wikipedia requires a fine balance: Too little time devoted to the process leaves students and faculty uncomfortable working in the online environment, yet too much time causes students to resent the focus shifting away from their studies. We believe the second iteration of the course has begun to correct this effort/output balance.
Future directions for this project include collaborations with others interested in EBM teaching using Wikipedia’s online platform. We are refining the design of this structured EBM editing project through ongoing field testing at Queen’s University, and we plan to examine the downstream effect on student behavior and knowledge. Additionally, we are exploring opportunities for students and faculty who make substantial improvements to articles to publish their work in Wikipedia’s online open access journal, the Wikijournal of Medicine. Working with the existing Wikipedia editor community before scaling up this type of initiative will be vital. Having online moderators/tutors who are trained to edit Wikipedia will ensure that the Wikipedia volunteer community is not overwhelmed with the task of verifying student edits and that the article improvements meet Wikipedia guidelines. Finally, while this project does not explicitly integrate individual patient values and preferences—a key component of evidence application—enhancing the quality of information on a freely accessible platform will facilitate the process of shared decision making.
Wikipedia effectively restructured the process of knowledge sharing by implementing a philosophy of crowd-sourced, egalitarian editing. Similarly, disruptive innovation in medical education can create transformative change in the process of acquiring and applying knowledge. Using a Wikipedia editing platform with a structured framework teaches medical students the process of EBM, and it has the altruistic side effect of improving an important resource that everyone can use. This authentic task adheres to the principles of high-quality instruction in EBM and could be used by a variety of health care educational initiatives.
The authors would like to acknowledge the members of the WikiEdu Foundation and WikiProject Medicine for their online support and guidance, especially for their student editors.
1. Dawes M, Summerskill W, Glasziou P, et al.; Second International Conference of Evidence-Based Health Care Teachers and Developers. Sicily statement on evidence-based practice. BMC Med Educ. 2005;5:1.
2. Albarqouni L, Hoffmann T, Glasziou P. Evidence-based practice educational intervention studies: A systematic review of what is taught and how it is measured. BMC Med Educ. 2018;18:177.
3. Maggio LA, ten Cate O, Chen HC, Irby DM, O’Brien BC. Challenges to learning evidence-based medicine and educational approaches to meet these challenges: A qualitative study of selected EBM curricula in U.S. and Canadian medical schools. Acad Med. 2016;91:101–106.
4. Khan KS, Coomarasamy A. A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Med Educ. 2006;6:59.
5. Gulikers JTM, Bastiaens TJ, Kirschner PA. The five-dimensional framework for authentic assessment. Educ Technol Res Dev. 2004;52:67–86.
6. Heilman JM, West AG. Wikipedia and medicine: Quantifying readership, editors, and the significance of natural language. J Med Internet Res. 2015;17:e62.
7. Murray H. More than 2 billion pairs of eyeballs: Why aren’t you sharing medical knowledge on Wikipedia? BMJ Evid Based Med. 2019;24:90–91.
8. Matheson D, Matheson-Monnet C. Wikipedia as informal self-education for clinical decision-making in medical practice. Open Med J. 2017;4:15–25.
9. Azzam A, Bresler D, Leon A, et al. Why medical schools should embrace Wikipedia: Final-year medical student contributions to Wikipedia articles for academic credit at one school. Acad Med. 2017;92:194–200.
10. Finley AP, Rhodes TL. Using the VALUE Rubrics for Improvement of Learning and Authentic Assessment. 2013.Washington, DC: Association of American Colleges and Universities.