Secondary Logo

Journal Logo

Perspectives

Communities of Practice in Peer Review: Outlining a Group Review Process

Nagler, Alisa JD, MA, EdD; Ovitsh, Robin MD; Dumenco, Luba MD; Whicker, Shari EdD; Engle, Deborah L. EdD, MS; Goodell, Kristen MD

Author Information
doi: 10.1097/ACM.0000000000002804
  • Free

Abstract

The gold standard for scientific scholarship is publication in a peer-reviewed journal. To complete the peer-review process, most academic journals rely on a large group of individual volunteer reviewers—experienced professionals who review and provide feedback on manuscripts. Although these reviews undoubtedly improve the published end product, they are dependent on individuals with diverse expertise, experience, time, motivation, and editorial training. This is a challenge inherent within the current peer-review process. While there are resources available in the literature for self-training, a mechanism through which reviewers can actively learn effective strategies from others with more advanced or differing expertise is lacking. This can result in an intimidating and isolating peer-review experience. Moreover, the current process of individual peer review does not align with the collaborative team approach that is being incorporated throughout medical practice, education, and research.1,2 We suggest that the peer-review process would benefit from better alignment with this philosophy.3

Medicine has long been considered a community of practice (CoP).4 Lave and Wenger’s model of situated learning5 proposes that learning itself involves a process of engagement in a CoP. CoP members interact regularly, improving skills and increasing knowledge through peer mentorship. Members also have the opportunity to promote innovation and develop social capital. Using a group process to review manuscripts exemplifies application of this model, wherein individuals work in a community to successfully review manuscripts and to further develop their reviewing skills.3,6

The Journal of the American College of Cardiology has developed a mentoring program in which fellows perform independent reviews, followed by presentation to a group of editors for feedback to develop peer-reviewing skills.7 Similarly, an emergency medicine blog has implemented a coached peer-review model through a series of online interactions with an editorial team for training of individuals submitting posts to the blog.8 Others have suggested that junior scholars could benefit from a “team-based coaching approach” to peer review.9 More recently, several medical education journals6 have begun to include an option for group peer review (GPR). Academic Medicine (AM) includes a statement encouraging reviewers to “mentor junior faculty on the peer-review process. If you wish to involve a specially qualified colleague in the review, you must contact the editorial office and ask for permission ahead of time.”10Teaching and Learning in Medicine recently added a note to invited reviewers, welcoming them to mentor or invite group peer reviewers, “which can help reviewing serve both community- and self-development purposes. . . . It is also an enjoyable way to collaborate with peers.” Despite these developments, there remain several unanswered questions regarding GPR, including the advantages and disadvantages compared with individual peer review, as well as the optimal functional approach to GPR.

We developed a GPR process which enables reviewers to (1) learn and receive real-time feedback from one another, acting as peer mentors and enabling a group of reviewers from diverse backgrounds and levels of expertise to learn from varying perspectives; (2) enhance engagement and satisfaction with the review process; and (3) provide more robust feedback to journal editors and authors as a result of a broader scope of expertise.

Implementing GPR

In November 2015, our group of 6 medical educators met for the first time at a workshop conducted by the AM editorial staff at the Association of American Medical Colleges Learn Serve Lead meeting in Baltimore, Maryland. At this workshop, intended to train individual reviewers, table groups worked together to review a manuscript. Our group found the process to be enjoyable and high yield for our own development, mostly as a result of the shared expertise and group discussion. We aimed to continue working together to further explore the benefits of GPR and to investigate how GPR might best be operationalized.

Through a series of emails and conference calls, the 6 of us agreed to write up our initial experience and formally propose GPR as an innovation in medical education scholarship. Once this work was published in February 2017,3 we partnered with AM and began formally reviewing manuscripts as a group, with 3 major goals: to refine and codify an effective process for GPR, to provide robust feedback to the editors and authors, and to further develop and broaden our individual skills in peer review.

During our initial calls, we spent extensive time sharing individual areas of interest and expertise and identifying 8 topic areas (from a list supplied by AM) that the group was willing to review. Since we began reviewing, we have had 8 conference calls ranging from 55 to 120 minutes to finalize reviews for submission and four 30- to 60-minute calls to discuss process and next steps. Each call was scheduled by email, using Doodle polls. We tracked the time spent preparing for the call, engaging in the discussion, and finalizing feedback and recommendation documentation; after each review, we documented our reactions to the process. Over the year, we have developed a set of principles that define our process and group norms.

Our iterative approach has resulted in a 10-step process that we believe is an efficient and effective means for reviewing manuscripts as a group. Specifically, this process can lead to personal satisfaction and professional development for reviewers in addition to providing more robust and useful feedback.

  1. One group member is identified as the liaison with the journal (in this case, AM). The liaison receives the review request from AM and shares it with the group for a vote on its appropriateness for the group review based on the collective interest and expertise.
  2. If agreed upon, the manuscript is circulated to all group members.
  3. Group members each identify 2 sections of the manuscript for special focus, distributed such that each section is reviewed by at least 2 individuals. Overlap in sections is intentional and helps provide multiple perspectives on the same section. A review call is then scheduled (typically 2–3 weeks out).
  4. Group members each read the entire paper and record their overall impression of the manuscript, including preliminary publication recommendations (accept, resubmit with major/minor revisions, or reject). In addition, each member takes detailed notes on her particular section.
  5. Immediately before the group call, each member submits her individual preliminary recommendation (via an online survey) for later “reveal” and discussion.
  6. At the beginning of the call, each group member verbally provides her overall impression of the paper, in 2 minutes or less. Then, each section of the paper is discussed in detail, each individual facilitating the discussion of her designated sections. One or 2 members take notes during the discussion using a template on Google Docs for all to see electronically and comment on during the “meeting.” This becomes the first draft of what will be submitted as our formal review.
  7. At the end of the call and discussion, each member notes whether her individual preliminary recommendation changed following group discussion and elaborates as to why. This is documented and tracked. The final publication determination is deliberated on until there is a consensus decision and the recommendation to the journal’s editors can be finalized.
  8. Following the call, comments are finalized by one member using the Google Doc notes taken during the call and then circulated electronically in their entirety for all group members to review and edit.
  9. The journal liaison then transfers the comments into the AM online review form and completes all other requirements for reviewer submission.
  10. The journal liaison notifies all group members of the submission and, ultimately, the journal’s final publication decision.

What We Have Learned

In addition to developing a clear process for GPR, our group devoted significant time to clearly elucidate the benefits and limitations of this process.

We aimed to safeguard against the risk that “groupthink” could lead members to change their recommendations, by building in a process for the initial collection and undisclosed recording of each member’s individual initial recommendation (Step 5). This is followed by a group discussion on why each individual made her decision, with each individual afforded the time to clarify and expound on her own informed professional opinions (Step 7). This process encourages “diversity of thought” and negates the tendency toward groupthink.11 The results of Steps 5 and 7 are illustrated in Table 1. The group agreed on “revise and resubmit with major revisions” for 3 articles and “reject” for the fourth. There is no obvious pattern to what each individual recommended initially and how their recommendation changed following the GPR discussion; however, our current sample size is small.

Table 1
Table 1:
Outcomes of 4 Academic Medicine Articles Reviewed by 6 Reviewers Engaging in a Group-Peer-Review Process, 2017–2018

We believe that this open dialogue and collaborative discussion allowed everyone to voice her initial ideas and learn from each other’s areas of expertise. This process helped us to determine the most appropriate recommendation following a very thorough group review and shared contributions. We do not yet know whether the overall GPR process results in “more stringent” recommendations given our current limited data. However, based on our experience with all 4 manuscripts, 50% of reviewers changed their individual publication recommendation to a more stringent recommendation category, 12.5% changed to a less stringent category, and 37.5% stayed the same. In addition, as can be seen in Table 1, the results of our GPR were more stringent than those of 2 other individual reviewers, less stringent than those of 2 other individual reviewers, and the same as those of 3 other individual reviewers. Compared with decisions by the editor, our recommendations were aligned for 3 of the manuscripts, and for the fourth (Article #3), we recommended “major revisions” and the editor “minor revisions.” While it is too early to determine whether GPR leads to more stringent recommendations overall, it is important to note that any recommendation (more stringent or not) that includes robust feedback and supporting evidence should be considered a positive one and viewed favorably.

Another way we have evaluated the impact of GPR was by tracking per-person review time. Table 2 illustrates the average per-person time allocated to each step of the process, as well as aggregate person-hours. We became more efficient over time, moving from a person-hour total of 34 hours to 17.75 hours, from our first GPR to our fourth review. Over time, we spent less time developing and refining the process and identified and implemented ways to save time on the actual review (e.g., using a shared GoogleDoc to take notes during the group call so as to more easily convert the notes into our final feedback to the author and editor; identifying the liaison to the journal with a simple rotation). Through this process, we also were able to identify individual member strengths and, as a result, assign sections of each manuscript to the individuals who would be most efficient and effective at reviewing those sections.

Table 2
Table 2:
Time Spent by 6 Reviewers Using a Group-Peer-Review Process to Review 4 Manuscripts Submitted to Academic Medicine, 2017–2018

As can be seen from Table 2, by the fourth review, the average total time spent per person was less than 3 hours. There is not a great deal of literature regarding the average amount of time spent on independent reviews, likely partly because this varies widely by journal specialty.12 However, the process is typically time-consuming12,13 and has been reported to take a reviewer anywhere from several hours to days per review.12,13 Those of us who already independently review for AM have estimated that we would have spent a similar amount of time as each of our 4 group reviews if they had been done independently. With GPR, the product, individual satisfaction, and potential gain are so much greater.

We recognize that with 6 individuals contributing, the total number of aggregate person-hours is significant. The next phase of this project will involve comparing the quality and benefits (to both authors and journals) between individual and group review processes. When those data are available, we may be able to perform a cost–benefit analysis. In the meantime, we believe that each of us would have spent a similar amount of time if we had reviewed independently, and there are additional benefits to this process compared with independent review. These include the benefits to the reviewers (who experience enhanced enjoyment and gain increased skill from working with colleagues) and, potentially, to the authors and journal editors, although these aspects remain to be studied further. It is clear, however, that obtaining high-quality peer reviews in a timely fashion is challenging for journal editors.12 While many journals seek input from 2 to 3 external individual peer reviewers, the average number of referee reports in medical journals is 2.2.12 With the increasing number of scholarly publications, it has been reported that some journal editors may now need to send out 10 to 12 invitations or more to capture 1 to 2 reviewers.14 In addition, the peer-review process can be time-consuming and result in delays in publication, particularly if there is a lack of consensus in independent reviews.12 It is possible that established groups of peer reviewers could facilitate this entire review process by ensuring a much broader spectrum of expertise than independent reviewers. Finally, amongst our group, we each find ourselves more likely to agree to do a group review than an individual one, when time constraints exist, because of the added benefits. This finding is not surprising when one considers that studies have shown that reviewers value feedback on reviews over monetary rewards or payment in kind.14 The learning process and professional benefits in GPR provide high-quality feedback with a short turnaround time.

Beyond our quantitative outcomes, we have tracked our impressions and lessons learned as the process has evolved. These have resulted in a set of operating principles (List 1) for optimal group functioning. Additionally, List 2 describes several explicit benefits that our group has unanimously expressed as a result of our experience as group reviewers.

List 1

Operating Principles of Group Peer Review

  1. Prioritize inclusion of all group members in decision making (over the convenience of agreeing on a quorum and over deference to any one individual’s experience).
  2. Commit to rigorous scholarship (learning what we do not know, tracking down references, being diligent and thorough in our feedback to the author and editor).
  3. Share leadership and responsibility, rotating on an ad hoc basis depending on the needs and schedules of individual members.
  4. Maintain a flexible and iterative process.

List 2

Benefits for Reviewers of Group Peer Review

  1. Rapidly expanding individual expertise as we learn from each other through this “community of practice”
  2. Increased confidence in the comments that are ultimately submitted
  3. Improved skills in reviewing manuscripts, which may lead to becoming a more sophisticated individual reviewer and author
  4. Significant enjoyment and satisfaction with the review process itself
  5. New professional network for future collaboration

Finally, we found that our process also addresses some of the problems inherent to individual reviews. During our discussions, it became clear that individuals in our group have different areas of expertise in reviewing (e.g., group members with qualitative vs. quantitative research backgrounds; group members who focus on details of tables and figures vs. those who focus on the context). When we review individually, there is no mechanism for identifying or filling these knowledge gaps. When we review as a group, we readily identify our own individual areas of weakness and ask for clarification or input from others who are strong in these particular areas. This hopefully results in improved feedback to manuscript authors and journal editors (although this must be studied further) and leads to improved individual expertise and professional development.

The GPR professional development benefit is even more significant considering the paucity of training opportunities for improving the quality of peer review. While there are some online and in-person modules to learn how to review research papers initially, there is no formal mechanism in place for “continuing professional development” in peer review. Working together allowed us to mentor each other and to give and receive feedback. In this way, GPR supports lifelong learning and aligns with the most current team-based, collaborative models of medical education and practice, and research. For most medical educators, peer review is a task that happens alone, often during “free” time on evenings or weekends. It is intellectually challenging, but can be isolating, and often fits in the cracks between fixed work and personal obligations. Fortunately, GPR elevates the status of this challenging work and affords an opportunity for close collaboration and further professional development.

It is worth noting that current descriptions of a team-based coaching approach to peer review9 and “coached peer review”8 both address mentoring of junior reviewers in improving the peer-review process. While we believe the mentoring process to be extremely valuable, we believe that the GPR model can truly serve as continuing professional development at any stage of peer-review expertise (we did not have “junior” and “senior” reviewers in our group).

In addition, it should be noted that our group, which was formed serendipitously at a conference workshop,3 consists of 6 medical educators with varied backgrounds and areas of expertise. The optimal size and composition for GPR is not known. A team-based coaching model for peer review suggested that an experienced reviewer might invite “colleagues with topic and methodological expertise, and 4 to 6 interested junior reviewers per paper”9; however, there were no data presented to support this particular recommendation. Based on our experience, we believe that a group size of 4 to 6 individuals total would likely be optimal. This size would allow for diversity of expertise but would also mitigate logistical challenges that would likely exist with larger groups.

There are potential limitations of GPR. First is the aggregate increased time it may take to review in a group rather than individually, as demonstrated in Table 2. However, as alluded to above, the time involved for an individual reviewer in the group process is likely not greater than it would be for an individual traditional review. In addition, individual reviewers in the group are able to focus more deeply on specific sections of the manuscript (as identified/assigned). This may feel less overwhelming for the group reviewer and likely provides more in-depth feedback to authors on each section. In addition, all agree that even if it had taken more time, it is more rewarding personally and professionally to participate in the group review compared with working independently. Another potential limitation and a challenge we have faced is the difficulty in scheduling busy professionals. Sometimes this has delayed our final submission to the journal editor.

A final and significant limitation is that we cannot be sure that GPR, compared with individual peer review, indeed leads to better feedback to authors and journal editors. Through this process, however, the authors uniformly believe that the GPR feedback submitted in each case is much more robust and likely to markedly improve the quality of the publication compared with recommendations that each of us would have made independently. These observations are confirmed by our analysis of 2 group-peer-reviewed articles that have been recently published (see Table 1) and by analysis of the copy of the letter provided to reviewers that contains individual and group-peer-reviewer comments, as well as the final AM decision.

Despite these drawbacks, without hesitation, we feel that the benefits of GPR for the reviewers, authors, and journal editors outweigh the shortcomings.

Where Do We Go From Here?

There are multiple opportunities to enhance the GPR process and expand the reach of this project, for research and practical purposes. We plan to continue using our process shared above and refining as needed, as well as to encourage and support others in its use. As others adopt the process, we would welcome their feedback to further improve or enhance the process and identify any nuances that may arise based on team members, types of articles reviewed, live versus distance teamwork, organization of documentation, etc.

We plan to further evaluate the effectiveness of GPR. This will entail qualitative analysis after gathering feedback via survey from authors and journal editors, which may allow for a scholarly comparison of GPR with individual peer-review feedback. There is, however, literature to suggest that authors rate feedback more positively with positive review decisions,12 so valid author input may be challenging to obtain and interpret.

We also plan to survey journal editors broadly to assess the current use of the GPR option, elaborating on potential benefits and challenges. Finally, with a growth in the number of individuals engaged in GPR, we would like to gather feedback from group reviewers themselves. This may inform the group compositions and processes that are most effective (e.g., individuals from the same or different institutions, senior-level and junior-level team reviewer combinations, large or small groups, group members with varied experiences and areas of expertise). Findings will be shared and used to further refine and codify a GPR process that can be readily implemented at an individual manuscript level or formally incorporated and operationalized by journals.

There are additional ways to take advantage of the benefits of the GPR process. As described in our previous article,3 and proposed by others,6,9 institutions could conduct GPR workshops for junior faculty to help them become better reviewers and develop their scholarly writing skills, and to encourage them to write and review themselves. The National Institutes of Health has long relied on a GPR system in which selected scientific experts work together in scientific review groups to review grants and contracts.15 A GPR process could be implemented in additional grant-funding agencies as they consider which proposals to award. Senior and junior reviewers alike can contribute to the innovative landscape while simultaneously teaching and learning. Groups of faculty and/or learners could review already-published papers, as a modified journal club that focuses on writing and scholarship rather than the content of the research. Alternatively, authors could invite colleagues to review an article as a group before submission to gather robust varied feedback for purposes of making final improvements. Individual reviewers could also welcome others to join them in reviewing an article, as permitted by some journals. Editors may consider promoting GPR as they assign submitted manuscripts that might particularly benefit from this approach. On a grander scale, journals may consider holding GPR workshops (much like exam-item-writing workshops) as a means of efficiently receiving high-quality team reviews for a large number of submissions.

Conclusion

We propose that GPR is an innovative opportunity to engage in a scholarly CoP. This construct has great potential to improve individuals’ review skills, increase knowledge, make mentorship accessible, and increase social capital to all participants for future collaboration. Authors and editors will likely receive more robust feedback, and journals may be able to recruit and retain reviewers who favor this approach.

Acknowledgments:

The authors wish to thank Academic Medicine for supporting this work and especially Mary Beth DeVilbiss for sharing in the exploration of this innovative practice. They also wish to thank the staff of the Journal of Graduate Medical Education and authors of the following article for their related efforts and collaboration: Ilgen JS, Artino AR Jr, Simpson D, Yarris LM, Chretien KC, Sullivan GM. Group peer review: The breakfast of champions. J Grad Med Educ. 2016;8:646–649.

References

1. Burgess AW, McGregor DM, Mellis CM. Applying established guidelines to team-based learning programs in medical schools: A systematic review. Acad Med. 2014;89:678–688.
2. Leone Sciabolazza V, Vacca R, Kennelly Okraku T, McCarty C. Detecting and analyzing research communities in longitudinal scientific networks. PLoS One. 2017;12:e0182516.
3. Dumenco L, Engle DL, Goodell K, Nagler A, Ovitsh RK, Whicker SA. Expanding group peer review: A proposal for medical education scholarship. Acad Med. 2017;92:147–149.
4. Cruess RL, Cruess SR, Steinert Y. Medicine as a community of practice: Implications for medical education. Acad Med. 2018;93:185–191.
5. Lave J, Wenger E. Situated Learning: Legitimate Peripheral Participation. 1991.Cambridge, UK: Cambridge University Press.
6. Ilgen JS, Artino AR Jr, Simpson D, Yarris LM, Chretien KC, Sullivan GM. Group peer review: The breakfast of champions. J Grad Med Educ. 2016;8:646–649.
7. Cooper LB, Bellam N, Vaduganathan M; JACC: Heart Failure Fellows. Educating the next generation of peer reviewers. J Am Coll Cardiol. 2016;67:2079–2082.
8. Sidalak D, Purdy E, Luckett-Gatopoulos S, Murray H, Thoma B, Chan TM. Coached peer review: Developing the next generation of authors. Acad Med. 2017;92:201–204.
9. Yarris LM, Simpson D, Ilgen JS, Chan TM. Team-based coaching approach to peer review: Sharing service and scholarship. J Grad Med Educ. 2017;9:127–128.
10. Academic Medicine. Reviewer policies and FAQs. https://journals.lww.com/academicmedicine/Pages/ForReviewers.aspx. Published 2018. Accessed May 1, 2019.
11. Fernandez CP. Creating thought diversity: The antidote to group think. J Public Health Manag Pract. 2007;13:670–671.
12. Huisman J, Smits J. Duration and quality of the peer review process: The author’s perspective. Scientometrics. 2017;113:633–650.
13. Yankauer A. Who are the peer reviewers and how much do they review? JAMA. 1990;263:1338–1340.
14. Malcom D. It’s time we fix the peer review system. Am J Pharm Educ. 2018;82:7144.
15. National Institutes of Health. Grants and funding: Application referral and review. https://grants.nih.gov/grants/referral-and-review.htm. Published 2017. Accessed May 1, 2019.
Copyright © 2019 by the Association of American Medical Colleges