Communities of practice (CoPs) have been defined as “groups of individuals formed around common interests and expertise” that, when leveraged, can “solve problems [and] transfer best practices.”1 CoPs include a combination of both active and passive members and facilitate master–apprentice learning, whereby novices gradually increase knowledge and experience with the help of experts, often through passively watching or listening to experts, while experts gain knowledge through more active interaction with the community.2 Health care professionals are familiar with engaging in local CoPs within their hospital walls, region, and/or country, but despite the availability of online technologies that facilitate online global collaboration, the health care sector has yet to fully embrace these tools,3 which may be attributable to cultural preferences, privacy concerns, technical issues, perceived time constraints, difficulty in adapting these tools for health care professionals, and/or difficulty in engaging health care professionals with these tools.4
CoPs are heterogeneous in audience, scope, and goals, and thus, there is no standard validated framework for evaluating CoPs.2,3,5 Reported evaluation frameworks of medical CoPs include assessments of one or more of the following components of the CoP5:
- Goals/scope: description of what the CoP is seeking to accomplish;
- Stage of development: description of the CoP’s development stage (i.e., building, growth, adaptive, closed);
- Context: the external environment in which the CoP exists or the characteristics of the CoP itself (i.e., people, organization, system);
- Structure: description of the composition of the CoP, often using description measures (i.e., size, existence of subgroups) or social network analysis;
- Process/activities: description of CoP activities and ways the CoP intends to achieve its goals;
- Outcomes: positive or negative results of the CoP from an individual, group, or organizational level; and
- Level of impact: value or potential value of the CoP to the individual or entire group.
Studies most commonly report these components by describing site activity information and member surveys or interviews.2,5
In this report, we describe our innovative approach to initiate a global, online pediatric critical care CoP through an activity that combines both traditional in-person educational conferences and online interaction, and we evaluate it using the above components (we have listed the components in parentheses where we report on the corresponding information, with the exception of the outcomes, which are reported throughout the Outcomes section of this report).
OPENPediatrics6 is an online social learning platform developed at Boston Children’s Hospital in collaboration with the World Federation of Pediatric Intensive and Critical Care Societies and the IBM Corporation. Released in September 2012, OPENPediatrics has acquired over 7,443 registered users from over 1,800 hospitals in 132 countries as of November 2015. OPENPediatrics has three sites through which users can access information: a public open-access site, a secure clinician site, and a YouTube channel. On average, each month the open-access site receives 15,000 page views, the clinician site receives 2,000 log-in sessions, and the YouTube channel receives 10,000 page views.
OPENPediatrics launched the World Shared Practices video (WSP) series in March 2013 as a strategy to engage and coalesce the global community of critical care clinicians (CoP stage of development—building and growth). Each month, a well-known, international expert in pediatric critical care medicine is invited to be a speaker for the WSP series by a group of four faculty members in pediatric critical care medicine at Boston Children’s Hospital, all of whom are professors at Harvard Medical School and themselves experts in the field. The WSPs are filmed during the experts’ visits to Boston Children’s Hospital, and subsequently edited and reviewed by the speaker and one or two critical care faculty members before their release on the OPENPediatrics sites. Two to five questions pertaining to context and practice are interspersed into each 30- to 45-minute video, often with the speaker stating their expert opinion after posing the question to the audience.
Once released, a WSP can be viewed whenever it is convenient for the viewer. Viewers contribute to the community discussion by leaving comments that display alongside the video (CoP goals/scope—using the WSP as primary medium, community discussion is encouraged), without formal comment moderation. As CoPs are more likely to share knowledge when a combination of in-person and online activities are used,2 clinicians are encouraged to hold an educational conference at their hospitals so that they can watch the videos together, discuss responses to the questions, and submit a single group comment.
Conceptual framework of the WSP series
In a worldwide needs assessment conducted prior to developing OPENPediatrics, approximately half of respondents requested a question-and-answer forum (53% [228/429]), best practice webinars (48% [208/429]), and a place to network and talk with doctors and nurses worldwide (43% [186/429]).7 To address these needs, we conceptualized the WSP series. We chose an asynchronous format for the WSP series for several reasons. First, videos could be optimized for online viewing by incorporating engaging graphics, animations, and figures. Second, videos allow and encourage equal opportunities for participation worldwide, a goal that is unlikely to be accomplished by webinars presented at a set time which may be convenient for some but inconvenient for others. Although webinars can be recorded and archived for later viewing and discussion, in our experience, few clinicians return to watch archived webinars. Additionally, clinicians may be less likely to actively contribute to an educational opportunity for which a live portion has already been completed. Finally, having a scheduled release date and encouraging the incorporation of WSPs into their local educational conference schedule might enhance participation.
WSP release and global interaction
A WSP is released on the fourth Tuesday of each month, a recurring and predictable event that can easily fit into local educational conference schedules. While watching the video, the audience can respond to the interspersed questions posed to the global CoP, such as “How do you make the diagnosis of low cardiac output?” Viewers can see the responses from other participants and comment on their own practices (CoP context—global conversations about individual practices in pediatric critical care medicine) (see Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/A402).
Measures of community engagement and data analysis
We used embedded site analytics and Web-based surveys (see below) to gather the following data from March 2013 to November 2015 (CoP process/activities—metrics used to assess viewer engagement):
- The total number of video views and unique viewers, viewer names and roles, hospital names and locations, and comments on the clinician site were reported by IBM Cognos Analytics (IBM Corporation, Armonk, New York).
- The number of views on the public site and YouTube channel were reported by Vimeo (New York, New York) and YouTube (San Bruno, California) analytics. (As no log-in is required for these public sites, unique user information was not available.)
- The annual OPENPediatrics user experience survey was sent to all OPENPediatrics users on April 15, 2015, via SurveyMonkey (Palo Alto, California).
- A survey was sent to the viewers who commented on a WSP on the clinician site asking about WSP-viewing activity on November 19, 2015, via SurveyMonkey (Palo Alto, California).
We analyzed data using Microsoft Excel 2013 (Microsoft Corporation, Redmond, Washington) and SAS 9.4 (SAS Institute Inc., Cary, North Carolina).
Between March 2013 and November 2015, 28 WSPs were launched on a variety of pediatric critical care topics; these were viewed 18,414 times by 1,864 unique viewers in 132 countries and 760 hospitals. On average, each WSP was viewed over 247 times in the first 30 days following its release; 69% (4,748/6,928) of these views came from the clinician site, 24% (1,671/6,928) from the public site, and 7% (509/6,928) from the YouTube channel (see Supplemental Digital Appendix 2 at http://links.lww.com/ACADMED/A403).
On the clinician site, user engagement was most active during the first 30 days following a WSP release, with videos receiving almost half (4,748/11,109) of their lifetime views during this time. Log-ins to the clinician site were highest during the week prior to and the week of a WSP release, when release and save-the-date e-mails were sent to the OPENPediatrics distribution list. Approximately one-third (2,934/10,627) of the total clinician site visits during WSP release weeks included a WSP view. The average lifetime total views for each WSP on the clinician site were approximately 400 views by about 186 viewers (see Supplemental Digital Appendix 2 at http://links.lww.com/ACADMED/A403).
Twenty-five percent (1,864/7,443) of our registered users, a diverse global audience (CoP structure—the size and composition of our CoP) (see Table 1), viewed at least one WSP. Attending physicians/consultants, an audience that, in our experience, is often reluctant to engage with new technology, made up the largest group of WSP viewers at 36% (671/1,864).
Videos that covered sepsis, cardiovascular, and respiratory topics generated over twice as much participation and, on average, were viewed 130 times by 60 viewers each, as compared with videos on history and quality improvement, which, on average, were viewed 55 times by 30 viewers each (see Supplemental Digital Appendix 2 at http://links.lww.com/ACADMED/A403).
Eight percent (149/1,864) of WSP viewers on the clinician site contributed to community discussions and left 1,155 comments. On average, each video received approximately 40 comments from about 14 viewers on the clinician site (see Supplemental Digital Appendix 2 at http://links.lww.com/ACADMED/A403). After controlling for the number of viewers from each region, there was no statistically significant difference in the number of comments received by location (F = 0.89) (see Table 1). Forty-seven percent (876/1,864) of WSP viewers viewed at least two WSPs, and 15% (280/1,864) viewed five or more. There is a positive correlation between the number of comments submitted by viewers and the number of WSPs they viewed (Spearman correlation coefficient = 0.32, P < .001). Viewers that commented are highly engaged members in this global, online CoP, watching on average seven or eight WSPs and commenting half of the time, suggesting that there are some champions (or very engaged users). Of viewers who left comments who responded to our survey (81/149 from 40 countries), 37% (30/81) reported viewing the WSPs in small groups with an average of 8 clinicians. Although the number of actual commenters was small, reporting the number of comments is not an adequate metric to capture and reflect vigorous discussions that might be occurring locally when individuals in a hospital view such an online educational lecture together, as described in a local pediatric intensive care unit CoP in Canada.8 The small number of comments may also be indicative of the magnitude of health care professionals’ online engagement.9
Ninety-six percent (1,111/1,155) of comments were responses to the interspersed questions, and 4% (44/1,155) were general comments or questions. We thought that interspersing questions in the videos would stimulate discussions among clinicians locally and globally, and indeed, most of the comments were related to these questions. However, our hope that these questions would allow clinicians to become comfortable with commenting in an online platform and stimulate additional discussions over time was not realized. Instead, we observed a decrease in additional commenting over time. Possible explanations include clinicians not feeling comfortable asking additional questions when no other peers were doing so, clinicians feeling intimidated to ask questions of the expert WSP speakers, or perhaps additional questions were being answered in local discussions. To try to encourage ongoing discussion, the WSP speakers responded to questions for the lifetime of the WSP, and 48% (21/44) of general questions were answered by the WSP speaker or the speaker’s designee.
Although our study was not designed to assess impact in practice or clinical outcomes, our 2015 annual OPENPediatrics user experience survey (480/7,443 respondents from 84 countries) offered insight into perceived satisfaction and behavioral changes related to the WSPs (CoP level of impact—satisfaction with and change in knowledge or practice resulting from CoP engagement). Fifteen percent (56/376) of respondents reported that the goal of their last visit to OPENPediatrics was to watch a WSP. Ninety-six percent (310/323) of respondents reported that the WSPs were valuable or very valuable to their learning or teaching, and 15% (21/144) of respondents reported that the WSP series led to a positive change in their knowledge or practice.
Twenty-one percent (1,107/5,226) of viewers on the clinician site watched an entire video in a single sitting (see Supplemental Digital Appendix 2 at http://links.lww.com/ACADMED/A403), which is much higher than expected for the approximately 40-minute videos. A recent study by the online education site edX about student engagement with online math and science videos found that watching an entire video declined rapidly when videos were more than 6 minutes long.10 The difference seen on our clinician site was likely due to our audiences having different motivations for viewing the WSPs. We used well-known experts as speakers, likely increasing interest. Also, scheduling a structured time to watch together in a traditional educational conference may have helped promote longer view times. Finally, clinician viewers likely found the videos relevant and valuable to their practice, as supported by the number of viewers that returned to watch more than one WSP and reported satisfaction with the WSP series (see above).
Our study has some limitations. One limitation is the difficulty in providing an evaluation of our CoP that allows us to benchmark our work against other CoPs. This is related to the heterogeneous nature of CoPs and the lack of agreement on a standardized framework for evaluating CoP activities.2,3,5 Although we described our CoP using basic metrics similar to those reported in the literature, more work is needed to better detail the context and structure of our CoP and to more deeply investigate its higher-level outcomes and impact.
Emerging technologies hold the potential to enhance global collaboration in the field of medicine across clinical, education, and research domains. Through our novel approach with the WSP series, we have demonstrated the potential to leverage online technologies to coalesce a global, online CoP in a format that could be used to stimulate discussion and collaboration within other health professional CoPs. The success of this video series may be due to using well-known subject experts to increase interest, encouraging local group viewings, and providing an asynchronous format for convenience. WSPs are viewed by a large percentage of our OPENPediatrics users, who reported finding the videos to be valuable for clinical practice. It is often difficult to engage busy clinicians in online activities, but we found that there were some champions who participated in the WSPs regularly. Active discussions may occur in local CoPs but cannot be adequately captured by current metrics.
Our future research will focus on further describing the context and structure of our CoP and more deeply investigating its higher-level outcomes and impact through more formal structured qualitative evaluation strategies. We will also seek to identify barriers that limit and strategies that improve clinician engagement with online communities through online surveys and structured interviews. Using enhanced analytic capabilities, we will also aim to identify and report additional metrics from other social interaction tools on the OPENPediatrics site to better explore the social learning behaviors of our CoP.
Acknowledgments: The authors would like to thank the pediatric critical care community of practice, especially the hospital champions who organize the local conferences, for their engagement with the World Shared Practices videos (WSPs), both viewing and commenting. The authors would also like to thank the expert speakers who agreed to be interviewed and filmed for the videos and the entire OPENPediatrics team for their hard work, creativity, and attention to detail required to produce the WSPs, especially Annalise Littman for her medical video editing and Aron Willey, Karen Gerofskey, Brandon Moye, and David Humphreys for their expert production. Finally, the authors wish to thank the IBM Corporation for their in-kind support in developing the platform to host OPENPediatrics.