Secondary Logo

Share this article on:

Coached Peer Review: Developing the Next Generation of Authors

Sidalak, Daniel; Purdy, Eve MD; Luckett-Gatopoulos, S. MD, MSc; Murray, Heather MD, MSc; Thoma, Brent MD, MA; Chan, Teresa M. MD

doi: 10.1097/ACM.0000000000001224
Innovation Reports

Problem Publishing in academic journals is challenging for learners. Those who pass the initial stages of internal review by an editor often find the anonymous peer review process harsh. Academic blogs offer alternate avenues for publishing medical education material. Many blogs, however, lack a peer review process, which some consumers argue compromises the quality of materials published.

Approach CanadiEM (formerly BoringEM) is an academic educational emergency medicine blog dedicated to publishing high-quality materials produced by learners (i.e., residents and medical students). The editorial team has designed and implemented a collaborative “coached peer review” process that comprises an open exchange among the learner–author, editors, and reviewers. The goal of this process is to facilitate the publication of high-quality academic materials by learner–authors while providing focused feedback to help them develop academic writing skills.

Outcomes The authors of this Innovation Report surveyed (February–June 2015) their blog’s learner–authors and external expert “staff” reviewers who had participated in coached peer review for their reactions to the process. The survey results revealed that participants viewed the process positively compared with both traditional journal peer review and academic blog publication processes. Participants found the process friendly, easy, efficient, and transparent. Learner–authors also reported increased confidence in their published material. These outcomes met the goals of coached peer review.

Next Steps CanadiEM aims to inspire continued participation in, exposure to, and high-quality production of academic writing by promoting the adoption of coached peer review for online educational resources produced by learners.

Supplemental Digital Content is available in the text.

D. Sidalak is a medical student, McMaster University, Hamilton, Ontario, Canada.

E. Purdy is an emergency medicine resident, Queen’s University, Kingston, Ontario, Canada.

S. Luckett-Gatopoulos is an emergency medicine resident, McMaster University, Hamilton, Ontario, Canada.

H. Murray is associate professor of emergency medicine, Queen’s University, Kingston, Ontario, Canada.

B. Thoma is clinical assistant professor of emergency medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada.

T. Chan is assistant professor of emergency medicine, McMaster University, Hamilton, Ontario, Canada.

Funding/Support: None reported.

Other disclosures: Dr. Eve Purdy, Dr. S. Luckett-Gatopoulos, Dr. Heather Murray, Dr. Brent Thoma, and Dr. Teresa Chan are on the volunteer editorial team of CanadiEM.

Ethical approval: The survey described in this Innovation Report received an exemption from the Hamilton Integrated Research Ethics Board because it was deemed a quality assurance project. All efforts were made to adhere to guidelines set by the Standards for Reporting Qualitative Research.

Previous presentations: This work has previously been presented at the 2015 Canadian Conference on Medical Education, Vancouver, British Columbia, Canada, as a poster and at the 2016 International Conference on Residency Education, Vancouver, British Columbia, Canada, as one of the top five “What Works” oral abstract presentations.

Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A356.

Correspondence should be addressed to Teresa Chan, Room 264, McMaster Clinic, Hamilton General Hospital, 237 Barton St. E., Hamilton, Ontario, Canada L8L 2X2; e-mail: teresa.chan@medportal.ca; Twitter: @TChanMD.

Back to Top | Article Outline

Problem

Most emergency medicine (EM) residents read academic blogs.1,2 These blogs provide an accelerated, convenient option for reading about medical innovations and decision making, for accessing physicians’ and others’ personal reflections, and for accessing published literature on conditions and treatments. Blog popularity arises in part from flexible publication formats and open accessibility online.

Contributors to peer-reviewed academic journals have criticized the credibility of blogs and other open online educational resources, citing a lack of traditional prepublication peer review.3 The traditional peer review processes, while conferring credibility, pose barriers to learners interested in publishing.3–5 In particular, the potential for strongly worded and critical reviews, and the lack of open dialogue between authors and reviewers, may discourage new authors.6 Authors are sometimes left feeling bullied and without an avenue for recourse.7 We worry that authors, especially junior authors, may discard many hours of work upon receiving a rejection and/or not pursue academic publication and peer review in the future.

Still, because credibility was identified as a key component of perceived quality in a recent review,4 academic blogs have responded. The editorial staff at the prominent EM blog Academic Life in Emergency Medicine, for example, developed an open expert prepublication peer review process5 for its learning materials. Other blogs have followed suit.

Back to Top | Article Outline

Approach

BoringEM is a multiauthor EM blog founded in 2012 that has recently been incorporated into a larger blog, CanadiEM.org. CanadiEM creates content produced predominantly by medical learners (i.e., medical students, interns, and residents). In 2014, more than 350,000 viewers accessed BoringEM, and it is ranked 12th of 156 EM and critical care blogs using the Social Media Index, an impact metric for blogs and podcasts in the areas of EM and critical care.8 The CanadiEM editorial team comprises two senior editors (B.T., T.C.) and two junior editors (E.P., S.L.-G.). All four staff members have blog-writing and editing experience. The two senior editors hold peer review and editorial positions at other academic blogs, serve as peer reviewers for major EM and/or medical education journals, and have each published 30+ peer-reviewed articles.

Acknowledging that readers could perceive CanadiEM’s commitment to publishing material created by learners as a threat to credibility, the editorial team developed a collaborative prepublication peer review process called “coached peer review.” The purpose of CanadiEM’s innovative coached peer review process is to promote positive and professional relationships between authors and reviewers such that they work collaboratively to publish online medical education resources that are rigorously vetted and edited to enhance credibility. This process aims to meet the unique needs of its learner–authors.

CanadiEM’s prepublication coached peer review strategy emphasizes a nonjudgmental editorial process. The goal of coached peer review are to improve learners’ medical knowledge, as well as their writing and editorial skills, by providing learner–authors with feedback from attending physicians who serve as reviewers and editors. The focus of these interactions is to improve the quality of the article, rather than determine whether it should be accepted or rejected for publication. The CanadiEM philosophy is that all learner–authors’ submissions—given enough time, attention, and revision—are ultimately publishable. This learner-centered editorial philosophy encourages and supports fledgling writers and helps them develop component skills of the CanMEDS framework (within, specifically, the medical expert, scholar, professional, communicator, and collaborator domains).9

The coached peer review process (Figure 1) begins either when an editorial team member recruits a learner–author to write about a topic of interest, or when a learner–author contacts CanadiEM directly, citing interest in a particular subject. The editorial team guides the learner–author through topic refinement using an internally developed curriculum map. After the learner–author completes an initial draft, she or he can suggest an external reviewer (i.e., “staff reviewer”) who is an expert on the topic, or a member of the editorial team will assign such an expert. In addition to this external review, all articles undergo review by an internal editor. The internal editor is a junior or senior member of the editorial team and serves as a peer reviewer and copy editor. The internal editor reads for style, grammar, flow, and clarity. The expert staff reviewer is an attending physician who provides content expertise and clinical context. Once the external review is complete, editors, staff reviewers, and learner–authors send the draft back and forth amongst themselves until everyone is satisfied with the finished product. Finally, the article is published on the blog concurrently with an open commentary (dubbed “Reviewing with the Staff”) written by the expert staff reviewer.

Figure 1

Figure 1

Submitted pieces vary in length, depth, and initial quality, and needs for editorial coaching vary widely. Each mentoring experience entails a series of e-mails exchanged among learner–author, internal editor, and staff reviewer. Editors and staff reviewers provide learner–authors with individualized feedback during each round of reviews via tracked changes. Sometimes multiple sets of revisions and rewrites are required to ensure that the final product meets the expectations of internal editors before it can be sent to the expert staff reviewer. Internal editors aim to ensure not only the quality of the written piece but also that the time and knowledge of the staff reviewers are used effectively. All suggested revisions are reviewed and accepted by the learner–author who, alongside the editor and expert staff reviewer, plays an active role in preparing the draft for expert review and publication. Should disagreements occur, the editorial team and learner–authors resolve them through discussion and consensus building. Formatting is customizable to the needs of the individual articles and aligns with content delivery needs.

The identities of the internal editor and the expert staff reviewer are fully disclosed, and informal discussion is encouraged. A sense of collaboration and accountability develops between the learner–author and editing team members as they build rapport with one another.

We developed and piloted coached peer review in July 2014, and fully implemented it in August 2014. Between July 2014 and March 2015, 10 learner–authors authored 24 articles for our academic blog. Fifteen physicians participated as expert staff reviewers. We recruited the staff reviewers through online searches and our own professional networks.

Back to Top | Article Outline

Outcomes

Assessing outcomes

To assess the experience and satisfaction of learner–authors and expert staff reviewers, we conducted a mixed-methods survey of these two groups between February 16, 2015, and June 14, 2015. The study population included medical student and resident learner–authors and physician staff reviewers.

Survey questions appear in Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/A356. Three members of our editorial team (D.S., E.P., and T.C.) developed the questions and pilot tested them with the other two members (B.T., S.L.-G.). Questions focused on understanding participant demographics and contrasting the coached peer review process with the traditional peer review processes used by most medical journals and with other online educational blogs.

We e-mailed an anonymous survey link to the blog’s learner–authors and staff reviewers, offering no incentives for participation. The survey used in this Innovation Report received an exemption from the Hamilton Integrated Research Ethics Board, which deemed the survey a quality assurance project.

We made every effort to adhere to guidelines set by the Standards for Reporting Qualitative Research. Two authors (D.S. and E.P.) performed qualitative analyses of the narrative responses by identifying themes, using an interpretive descriptive approach. After each author reviewed the responses independently, we compared and discussed our findings. We resolved disagreements by consensus after consulting with a third author (T.C.).

Back to Top | Article Outline

Respondent demographics and satisfaction

Of those surveyed, 10/10 (100%) learner–authors and 12/15 (80%) staff reviewers responded. The overall feedback from learner–authors and staff reviewers was uniformly positive.

Most learner–authors (7/10; 70%) and staff reviewers (11/12; 92%) participating in the survey had been involved, either as authors or reviewers, in traditional peer review processes. Of these, 15/18 (83%) rated our coached peer review process as superior. Specifically, all seven learner–authors agreed that coached peer review was superior: 2 (29%) rated it “better,” 4 (57%) rated it “significantly better,” and 1 (14%) rated it “the best” compared with traditional peer review. Of the 11 staff reviewers, 8 (73%) rated coached peer review superior to traditional peer review: 5 of these 11 (45%) rated it as “better,” and 3 (27%) as “significantly better.”

Back to Top | Article Outline

Comparison to traditional peer review

When compared with traditional peer review, learner–authors and staff reviewers valued (1) the speed and ease of publication; (2) the friendly, collegial, and nonconfrontational review process; (3) the opportunity to improve the quality of publication instead of receiving an outright rejection; (4) the increased transparency relative to blinded peer review; and (5) the increased speed of knowledge dissemination relative to the production process of typical journals. The negative aspects of our editing process that learner–authors and external expert staff reviewers identified included (1) too many back-and-forth revisions between authors and reviewers; (2) too many reviewers/editors assigned to an author; and (3) copy-editing periods that were longer than anticipated.

Because of CanadiEM’s editorial stance (i.e., that all submissions are ultimately publishable), some learner–authors felt there were “too many back-and-forth revisions.” During our initial period (July 2014–June 2015), only one author declined to revise a post after a second round of revisions, so this perception from the learner–authors surprised us. We believe the sequential reviewing process may have contributed to the perception, held by some learner–authors and expert staff reviewers, of prolonged review. This perception may have originated from the 30% of learner–authors who did not have prior experience with traditional peer review.

Back to Top | Article Outline

Comparison to other academic blogs

When compared with peer review processes for other academic blogs, learner–authors and expert staff reviewers valued (1) an involved editing team that was invested in the final published product; (2) clearly defined editing processes; (3) improved confidence in creating the written piece with appropriate guidance; and (4) the fast response times from experienced reviewers.

Back to Top | Article Outline

Discussion

The outcomes we have reported above are congruent with the coached peer review goals of promoting collegiality, collaboration, and quality improvement.

A transparent, collegial, and supportive peer review model like ours has the potential to support the development of the next generation of scholars. Our expert staff reviewers and learner–authors report strong engagement in this purposeful collaborative process. Because our staff reviewers are geographically diverse, the coached peer review may facilitate professional networking, aiding learner–authors in establishing connections outside of their institutions. Although there are differences in the process and content for publishing in traditional academic journals compared with academic blogs, we hope that our process serves as a starting point for learners as they pursue traditional peer review avenues later in their careers. We hope that their positive experiences with coached peer review will encourage them to pursue publication in traditional academic journals with increased confidence.

The time investment required to participate in coached peer review makes it challenging to implement and scale. Although we did not solicit information about barriers to participation, it is possible that the substantial investment of time required and lack of academic credit will limit growth of such editorial processes.10 We hope that the benefits of improving the accuracy and quality of academic blog posts through this process adequately justify the increased time investment required. We have found that having champions at a number of geographical locations is helpful for the sustained recruitment of learner–authors and staff reviewers—and for expediting turnaround time for reviews. Having learner–authors identify their own coaches also facilitates the process because a natural mentorship relationship may already exist. The survey had a small sample size, and as the number of participants increases, challenges that were not identified within this sample may become apparent.

Back to Top | Article Outline

Next Steps

Future research goals include measuring, through blog quality scores, how the credibility of CanadiEM is affected as a result of the coached peer review process. We also plan to determine the educational value of the process for learner–authors, expert staff reviewers, and readers. Measures tracking the educational value of this process may include evaluation of the level of coaching required by returning learner–authors on subsequent submissions.

We hope to encourage the adoption of our coached peer review process by other academic blogs. We believe this will improve the credibility of online educational resources and provide additional outlets for the development of learners’ academic writing skills. We hope to determine—once better methods for measuring the quality of blog-based medical education resources are available—whether the coached peer review results in higher-quality resources than nonreviewed content. We will also explore the adoption of certain aspects of coached peer review in traditional peer-reviewed journals. Finally, we may be able to track our cohort of learner–authors and compare them with their peers to see if they were more likely to become prolific in their academic writing.

In sum, innovations in editing processes in the digital world, such as CanadiEM’s coached peer review process, can contribute to upholding rigorous academic standards while also facilitating learner engagement and comfort in the academic publication process.

Back to Top | Article Outline

References

1. Purdy E, Thoma B, Bednarczyk J, Migneault D, Sherbino J. The use of free online educational resources by Canadian emergency medicine residents and program directors. CJEM. 2015;17:101–106.
2. CORD EM Blog. Academia rising: The long voyage from the Lyceum to Twitter. 2015. https://cordemblog.wordpress.com/2015/08/18/academia-rising-the-long-voyage-from-the-lyceum-to-twitter/. Accessed March 17, 2016.
3. Nicholas D, Watkinson A, Jamali HR, et al. Peer review: Still king in the digital age. Learn Pub. 2015;28:15–21.
4. Paterson QS, Thoma B, Milne WK, Lin M, Chan TM. A systematic review and qualitative analysis to determine quality indicators for health professions education blogs and podcasts. J Grad Med Educ. 2015;7:549–554.
5. Thoma B, Chan T, Desouza N, Lin M. Implementing peer review at an emergency medicine blog: Bridging the gap between educators and clinical experts. CJEM. 2015;17:188–191.
6. Triggle CR, Triggle DJ. What is the future of peer review? Why is there fraud in science? Is plagiarism out of control? Why do scientists do bad things? Is it all a case of: “all that is necessary for the triumph of evil is that good men do nothing”? Vasc Health Risk Manag. 2007;3:39–53.
7. Off C, Douglas J. Peer reviewer tells female biologists their study would be better if they worked with men. CBC. 2015. http://www.cbc.ca/radio/asithappens/as-it-happens-friday-edition-1.3057294/peer-reviewer-tells-female-biologists-their-study-would-be-better-if-they-worked-with-men-1.3057301. Accessed March 17, 2016.
8. Thoma B, Sanders JL, Lin M, Paterson QS, Steeg J, Chan TM. The social media index: Measuring the impact of emergency medicine and critical care Websites. West J Emerg Med. 2015;16:242–249.
9. Frank JR, Snell LS, Sherbino J. Draft CanMEDS 2015 Physician Competency Framework—Series III. September 2014. Ottawa, Ontario, Canada: Royal College of Physicians and Surgeons of Canada; http://www.royalcollege.ca/portal/page/portal/rc/common/documents/canmeds/framework/canmeds2015_framework_series_III_e.pdf. Accessed March 25, 2016.
10. van Rooyen S, Delamothe T, Evans SJ. Effect on peer review of telling reviewers that their signed reviews might be posted on the Web: Randomised controlled trial. BMJ. 2010;341:c5729.

Supplemental Digital Content

Back to Top | Article Outline
© 2017 by the Association of American Medical Colleges