Why Medical Schools Should Embrace Wikipedia: Final-Year Medical Student Contributions to Wikipedia Articles for Academic Credit at One School

Supplemental Digital Content is available in the text.


Problem
The rise and spread of Internet accessibility has created an unprecedented resource for the dissemination of medical information, as well as an invaluable tool for health care providers and the general public alike. At the same time, opportunities for the rapid spread of misinformation or the misinterpretation of medical facts have never been greater. Clinicians must sometimes gently redirect patients who are convinced that they have some rare disease they have read about on the Internet. Although medical educators typically train students to address patient misperceptions in clinical practice, they are not leveraging clinician and medical student knowledge to improve the quality of the information patients and others find online.
Wikipedia is a free, online, multilingual encyclopedia that is continually and collaboratively created. Anyone with an Internet connection can edit its articles. One of the most frequently visited Web sites worldwide, it is among the leading sources of health information for medical professionals and health care consumers alike. 1 At the end of 2013, Wikipedia's medical content included over 155,000 articles written in 255 languages, supported by more than 950,000 references. 2 Despite its increasingly prevalent use as a medical information resource in clinical practice, clinical instructors and faculty members often dissuade medical students from using Wikipedia, citing concern for its perceived inaccuracies and lack of traditional editorial controls. 3 Although physicians and medical students are encouraged to contribute to traditional sources of medical knowledge (e.g., textbooks and journals), the idea of benefiting from active contribution to crowd-sourced resources such as Wikipedia remains the perspective of a minority or fringe group within the academic medicine community.
We believe that not contributing to crowd-sourced resources represents a lost opportunity for enriching medical students' learning and for disseminating more accurate, up-to-date medical information to Wikipedia's readers worldwide.

Approach
We created what is, to our knowledge, the first formal medical school course worldwide through which medical students actively work to improve Wikipedia's health-related articles. We expected enrolled students to hone their information retrieval and assessment skills, 4 practice communicating medical knowledge to an exceptionally broad global audience, 2 and expand their sense Abstract Problem Most medical students use Wikipedia as an information source, yet medical schools do not train students to improve Wikipedia or use it critically.

Approach
Between November 2013 and November 2015, the authors offered fourth-year medical students a credit-bearing course to edit Wikipedia. The course was designed, delivered, and evaluated by faculty, medical librarians, and personnel from WikiProject Medicine, Wikipedia Education Foundation, and Translators Without Borders. The authors assessed the effect of the students' edits on Wikipedia's content, the effect of the course on student participants, and readership of students' chosen articles.

Outcomes
Forty-three enrolled students made 1,528 edits (average 36/student), contributing 493,994 content bytes (average 11,488/student). They added higher-quality and removed lowerquality sources for a net addition of 274 references (average 6/student). As of July 2016, none of the contributions of the first 28 students (2013, 2014) have been reversed or vandalized. Students discovered a tension between comprehensiveness and readability/ translatability, yet readability of most articles increased. Students felt they improved their articles, enjoyed giving back "specifically to Wikipedia," and broadened their sense of physician responsibilities in the socially networked information era. During only the "active editing months," Wikipedia traffic statistics indicate that the 43 articles were collectively viewed 1,116,065 times. Subsequent to students' efforts, these articles have been viewed nearly 22 million times.

Next Steps
If other schools replicate and improve on this initiative, future multi-institution studies could more accurately measure the effect of medical students on Wikipedia, and vice versa.
of health care providers' roles in the Internet age. We designed our course with assistance from WikiProject Medicine, a volunteer group of experienced Wikipedia editors who seek to ensure that the general public and health care professionals have access to free, current, accurate, and understandable medical information in their own language.
Thus far, we have run the course four times: November 2013, April 2014, November 2014, and November 2015. Each cycle has begun with a two-day orientation (Table 1) during which we introduce students to Wikipedia's editorial tools, style, and standards. Additional didactic activities include reviewing guidelines for writing simplified English and strategies for locating and evaluating source material quality.
All students establish their own Wikipedia user accounts and choose a single health-related article to edit over the remainder of the course. WikiProject Medicine maintains a list of the most frequently accessed health-related Wikipedia articles, ranked by importance, and graded according to an article quality scheme that is applied to most Wikipedia Projects. 5 We encourage (but do not require) students to select a "top-" or "high-importance" article that has also been tagged as needing quality improvement. At the beginning of the course, students receive custom reports for their chosen articles that contain both a list of grammar and style errors and a quantitative readability score. These Acrolinx (San Jose, California) reports use specialized natural language processing software, which considers sentence length and over 100 grammar and style rules.
Detailed course structure and materials are publicly available under free license on our course Wikipedia page. 6 As we are not always certain, a priori, how much students can accomplish in a month, we encourage them to improve their selected articles "as much as feasible." Thus far, students have completed the majority of their editing work independently. Instructors (A.A. and J.H.) and medical librarians (E.W. and L.M.) provide intermittent encouragement via e-mail, hold weekly office hours, and schedule consultation as needed. At the end of the course, faculty provide students access to custom-built tools that allow comparison of summary statistics of each article on the first versus last day of the course. While presenting the final versions of their Wikipedia pages, all students share highlights of their accomplishments and challenges, describe lessons learned, and provide feedback for future iterations of the course.
This report summarizes the outcomes of our pilot initiative at the University of California, San Francisco (UCSF). The UCSF institutional review board reviewed this research and deemed it exempt from formal review.

Effect on Wikipedia articles
We compared the state of the Wikipedia articles before and after student editing using several approaches. Two UCSFaffiliated physicians (E.W. and J.D.M.) provided independent subjective ratings of article changes and reconciled their impressions by consensus. Wikipedians with experience rating medical articles but unaffiliated with the elective received blinded unedited and edited versions of each article to evaluate using the general WikiProject article-grading rubric. We obtained changes in word, paragraph, and citation counts using Wikipedia's analysis tools. Finally, we quantitatively assessed each article's quality, readability, and translatability by generating postcourse Acrolinx reports.
Conveniently, all Wikipedia edits are saved in perpetuity. This archiving permits easy comparison across article versions over time. Because students edited their articles via their Wikipedia user accounts, we are able to analyze the changes (to words, paragraphs, and citations) made directly by our students. However, because anyone at any time can edit Wikipedia articles, all other metrics comparing the state of an article at the beginning and end of the course inevitably include changes made by other (nonstudent) editors.

Effect on medical students
At the midpoint of each course, three authors (A.A., E.W., L.M.) conducted one-on-one semistructured interviews with all enrolled students, asking nine questions about students' evolving impressions of the work of editing Wikipedia. Additionally, for each cycle we conducted a single end-of-course focus group with all actively enrolled students. All interviews and focus group sessions were audio recorded and transcribed. Several authors (A.A., A.L., L.M., K.O.) coded the emerging themes independently and then generated an overall theme list by consensus. Although we did not formally member check the final themes, we informed students that their comments would be deidentified and analyzed in aggregate for scholarly dissemination.

Effect on Wikipedia articles
During their participation in the course, the 43 students who enrolled (2013-2015) each edited a different Wikipedia article. Collectively, the students made 1,528 edits (average 36/ student), contributing 493,994 content bytes (average 11,488/student). They added higher-quality and removed lowerquality sources for a net addition of 274 references (average 6/student).
Additionally, we conducted a detailed analysis of the first three cycles of the course (November 2013-November 2014), during which 28 students enrolled and completed the course. Supplemental Digital Appendix 1 (http://links.lww. com/ACADMED/A386) provides a representative example of students' work. Subjective quality improvement ratings ("very improved," "improved," "unimproved," "worse," "much worse") determined by consensus between two UCSF physicians revealed that overall quality of 14 articles was "very improved," 12 articles "improved," and 2 articles "unimproved." Using the WikiProject article-grading scheme (stub, start, C, B, good, featured), all articles either improved in rating or remained unchanged; no articles declined in rating. Eight articles were initially tagged "Stub" or "Start" quality. Of these, 3 moved up two levels, 4 increased one level, and 1 remained unchanged. Fifteen articles were initially "C" quality. Of these, 4 increased one level, and 11 remained unchanged. Five articles were initially "B" quality, and all 5 of these remained unchanged. Table 2 summarizes the effect on Wikipedia article quality.
The quantitative review of individual student contributions revealed that 19 of these 28 articles had substantial changes to at least half of the sections. The average net change in number of paragraphs during the course was +7 (range −13 to +30). Three articles had a net deletion of words (−413 average; range −40 to −740). The remaining 25 articles added a total of 7,510 words (average net +300 per article; range +29 to +3,649). The number of citations increased in 24 articles. Among these, we noted an average net increase of 5.7 references per article (37.8 before to 43.5 after).
Natural language processing software showed modest but consistent average improvements across all scoring categories (all graded on a 100-point scale) for all 28 articles. In particular, overall quality-of-writing score improved in 23 articles (mean score increased from totals during the months each student was actively editing while enrolled in the course. b WikiProject Medicine quality scale is "stub," "start," "C," "B," "good article," "featured article." c Scale is a Likert-style list as "very improved," "improved," "unimproved," "worse," "much worse." 61.1 to 62.9), style score improved in 22 articles (mean score increased from 32.0 to 34.7), readability score improved in 22 articles (mean score increased from 64.9 to 66.5), and grammar score improved in 20 articles (mean score increased from 90.8 to 91.6).

Effect on medical students
The transcription and coding of interviews and focus groups with the first 28 students revealed several themes (see Table 3). Many students found researching and editing the articles to be more challenging than predicted. Some students cited the opportunity to give back to Wikipedia as a reason for enrolling, but quickly realized the work included much more than just writing. Students discovered a tension between comprehensiveness and readability/translatability, as the need to simultaneously address both general population and medical professional audiences proved difficult. Students had not anticipated and were at times challenged by the collaborative nature of the work, which variously required a willingness to rewrite, reorganize, or remove large portions of others' work. Students largely felt that these challenges were balanced by their perception of the utility and potential global significance of their efforts. Lastly, students deeply appreciated the independent nature of the course-allowing for flexibility and travel during their residency application process.

Discussion
The changing role of the provider. Students said that this elective widened their perceptions of being a health care provider in the modern age, especially in regard to Wikipedia. One of the more difficult facets of medical practice is effectively communicating complex health-related concepts. Although students have traditionally learned this skill "on the job," our course directly fosters clear communication in a nonclinical setting. Students gained concrete experience explaining nuanced health information in a way that is accessible to the general public. In addition, students reported that they felt joining the active community of Wikipedia editors and providing accurate information to communities that do not have access to other resources were rewarding. Furthermore, as the medical community moves toward more interprofessional care of patients and increases its reliance on electronic health records, this asynchronous crossdisciplinary virtual collaboration mirrors the evolving experience of clinical care. Additionally, by commissioning experienced Wikipedia editors as well as UCSF-affiliated physician-volunteers as participants in the course, we leveraged resources that have been untapped and/or underused in medical schools.

Meaningful and lasting improvements.
The students' positive feedback suggests that we have tapped a historically underdeveloped and potentially transformative opportunity to improve the curriculum. We believe that the wide variability across students' accomplishments in text and citation changes indicated not only each student's individual approach to editing but also, likely, the varying initial quality of the articles. In course feedback sessions, students frequently cited the flexibility to develop their own approach to editing the articles as a major strength of the course; however, this flexibility, paired with the variability in initial article quality, introduced challenges in evaluating students' efforts. For example, an article might be substantively improved through the removal of content, ultimately resulting in a decreased word count. Regardless, through July 2016, all of our students' contributions have persisted in Wikipedia-in several cases for over two years. Even marginal improvements thus can have a lasting impact.
According to Wikipedia's traffic statistics, during only the month that the students were actively contributing, the 43 articles edited in 2013, 2014, and 2015 were collectively viewed 1,116,065 times. Since then, between the end date of each course and October 31, 2015 (an arbitrary end date), these 43 articles had been viewed in aggregate 12,865,783 times. This remarkable total does not even include views or access via mobile devices (we do not have specific reports from mobile devices). Extrapolating Wikipedia traffic ratios of mobile versus desktop computer access in October 2015, 2 we estimate the overall views of these 43 articles since students finished editing until October 31, 2015, to be 21,992,791.

Evolving definition of scholarly work
As more journals move towards open access, and more health care professionals contribute to Wikipedia as the quintessential open resource, our students' efforts raise interesting questions about the definition of what constitutes scholarly activity within medicine and, more broadly, all professions. How does the creation of a community resource differ from the use of a textbook written by experts in the field? Should contributions to a community resource be considered a form of scholarly activity? Could viewership serve as a metric, complementary to impact factor? If a Wikipedia article edited by a medical student garners over 100,000 views/ month, might those edits constitute the greatest contribution to the medical literature in that student's nascent career? Are the public reviews, offered by professionals (or professionals-intraining), on talk pages of Wikipedia articles, serving as a version of peer review? And as professionals join the community of Wikipedia editors and contributors, how do their articles and edits change the definitions of "peer" and especially "peer review"?

Other metrics
In designing our first-of-its-kind course, we had no guidance on what outcome metrics to assess, so we made our best guesses, including counting visits to each student's Wikipedia article. We are not suggesting that more people go to these Wikipedia pages because they were touched by medical students; rather, we are showing that these numbers are significantly higher than those who typically assess traditional course assignments. In hindsight, these metrics prove only what we all intuitively know-a lot of people go to Wikipedia for health information. These data put that conventional wisdom into staggering context. Still, other innovative metrics may help us assess students' edited articles in the future.

Future studies
Our preliminary data are from a relatively small number of students; future studies should examine the effects of greater numbers of medical students' contributions to Wikipedia. Additional  Learning a lotmore than just writing • The first two days especially were much better than I expected. I learned a lot about how influential Wikipedia is.
• I don't feel like I ever really had the chance to do any digital work in med school. So I feel like this has been a good chance to … learn how to navigate this world.… Everyone should know how to do digital editing, and this is a good opportunity to do that.
• I sort of came in blind. I didn't really know anything about Wikipedia. I think it's really cool and I understand the idea behind it and I have so much more respect for what it is now-and with how the whole thing and community works. It's just awesome. Feels good to contribute to Wikipedia • I'm really learning how to meaningfully make an impact on WP which is a resource that I know I use very, very frequently.
• I like the idea of having WP, a source that people can go to without having subscription fees, and it is just a wealth of information that you can share.
• I just learned a lot about what Wikipedia is doing both for medicine and also for information for third world countries and areas where information could not easily be obtained. It just made me realize how important WP actually is. Before that I thought, it would be cool to help other medical students, other professionals through this resource. But now, I … realize that anyone could read the articles and they could be useful for anybody. • This has been an illuminating experience, to see how much thought and work goes into producing a product that is very easily consumable and almost disposable.
Why did you sign up for the course?
Schedule flexibility • I wanted a sort of travel-friendly elective.
Curiosity/learn more about Wikipedia • I was also interested to learn more about Wikipedia as well.
• WP is used so frequently and people do not give much thought to the amount of work that goes into it especially for a very high-quality article.
Explore topic of interest • Because one of my objectives … was to learn about a topic which will be useful for my future. And I feel like so far I have been learning details about my article which I wouldn't have taken the time otherwise to learn.
• When I first looked at the article, I realized very early that it was going to be a huge amount of work and so I thought it would be a good one to kind of increase the articles rating for.
Opportunity to write • Personally I really enjoy researching and writing.

Social impact of work
• I … think about the Wikipedia community and how many people are reading WP. And how I'm just one of many people working on this. And I just want to have a small role in making this better.
• If we are thinking about using this as a resource for people in Africa, or something like that where the only resource they can access is WP because it is the only thing that does not charge for data, then they should be able to find an explanation of … everything.
Has anything surprised you in your work so far editing your article?
Harder than expected • I had to basically rewrite the whole thing from scratch.
• I am not particularly skilled with technology, so I think actually it took a while after the orientation to get comfortable doing the edits.
• Going into the elective, okay, I had no idea of how hard it would be. Four weeks is a lot of time. I felt like I could [edit] a couple of articles in that amount of time. Once I learned how rigorous it was going to be to do the research, to do the sourcing, do the writing-it just made me realize how much work it was going to be.
• Now I am aware that a lot of work goes into editing. Each single sentence takes a lot of work.
• I think editing the article has not really been the more difficult part of the process so far. It has been … getting used to WP and the community, and using the proper syntax and editing in the actual WP site.
Easier than expected • I think I was expecting it to be harder to code … in WP, and it was actually a little bit easier.

High Wikipedia community engagement
• It has really been eye-opening to see how many people are involved in this process and how quickly it can be noticed [that] I changed something, even if it is a very minor edit.
• When I made a change and how quickly other people in the Internet world gave me feedback.
• I went on the talk page, and there is a section on the symptom and I kind of introduced myself and said I want to consolidate this list a little more, make it a little easier to read, and I wanted to see what other people thought about that. I had a couple of responses encouraging me to do that; they thought it would be a helpful change.

Low Wikipedia community engagement
• Yeah pretty much, he is the only one who has been engaging.
( Focused on leads/ start with simple things • I started by looking at the intro paragraph. The first few sentences were really long and I would imagine difficult to understand for a lay person, and I started by cleaning that up and just making it a bit more easy to chew on.
• I am making the definitions a bit more clear and then getting a reference for each one, as many as I can.
• I wanted to get the signs and symptoms down first, and also I have been working on the pathophysiology, and I have not put that section in WP yet but it is just about done.
Searched specific sources • I … looked at three or four different review articles just so I could get a consensus of what the literature said.
• I tried to find review articles but it seemed like most of the good information was in book chapters.
• For each subsection I would look it up on PubMed and see if there are any papers which [have] come out, which has been for the most part unsuccessful-at least for that section. So my second best is to go to UpToDate and see their list of signs and symptoms and see what papers they have referenced, trying to go through those papers and see if any of those papers are primary vs. secondary sources and see whether I will be able to use them. And then the next step is looking through AccessMedicine for the particular sign or symptom I am looking for and going to a specific textbook for that.
Hesitant to remove other people's work • There is a kind of permanence or finality with posting the changes to the article which concerns me and that is holding me back a bit.
• I definitely think about the work that someone has put into this and I certainly don't find myself highlighting and deleting full paragraphs because I think … that it is a pretty well-written page and obviously a lot of work has been put into it.
• It just seems so final to put something on the Internet. I know you can just edit it really easily, but I think it has taken me a little while to jump from doing the research and the lit review aspect to actually putting it on the page.
Too little vs. too much detail in some sections • I kind of approached it from the theory that as lopsided as the article is I was coming at it from the other side.… For me, it was done trying to strike a meaningful balance and to really represent those … there are multiple sides to it.
• I am interested in finding the signs and symptoms which are found in most people and then rearranging it in order of commonness.
• I think you could get lost in [editing] because you're kind of like, "How much do I put in this article that is part of the subsection but is referring to a full article that I think is not complete either?" • Generally you write something, you want to go for completeness and you want to make sure you covered everything. I think this is one of those cases in which people do not want to read something super-long; you want to hit the main points.
Making accessible for general audience • I think you can put what a medical person needs to say without saying it in a way that a normal person couldn't understand.
• I tried with my nondoctor hat on just to think about what it would sound like to read this whole sentence for someone who knows nothing except for the word that led them to the page.
• We want to focus on creating writing that is in the simplest form it can be in. The most understandable to everyone with various levels of literacy. And on the other hand, talking about how we want to make this a resource that is accepted within the medical profession. And something that doesn't [have] to be stigmatized. Oh, you know, I learned about this on Wikipedia but I'm not going to say that out loud because my attending wouldn't approve of that. We want to make it something that is scholarly work. And in that regard I find myself wanting, you know, to use medical terms and use the jargon because that is what makes it feel scholarly to me. And as a medical student that is what I would want to read. And so I felt like one of the biggest challenges is trying to reconcile those two things.
Abbreviations: UCSF indicates University of California; WP, Wikipedia. With the spread of similar courses, 7,8 greater numbers of medical students contributing to Wikipedia could create opportunities for multi-institutional collaborative studies with larger sample sizes. These future collaborations would more accurately measure the potential effect of medical students' contributions to Wikipedia-and vice versa.
The collaborative structure of Wikipedia creates an ideal platform for students across a range of health-related fields and geographic locations to work together to improve one of the most widely accessed repositories of health care information. 2 The nonprofit organization Translators Without Borders is actively partnering with WikiProject Medicine to translate health-related articles to other language Wikipedias. 9 As of July 2016, they have translated 637 English Wikipedia articles into 50 other languages. Furthermore, the Wikipedia Zero initiative now provides access to Wikipedia for free to over 600 million people in 57 developing countries via 75 mobile carriers. 10 Given the broad-ranging readership of Wikipedia, the potential for the next generation of health providers-working as "digital contributors" and not merely "digital consumers"-to effect positive worldwide change in human health is immense.