Secondary Logo

Journal Logo

Articles

From Modules to MOOCs: Application of the Six-Step Approach to Online Curriculum Development for Medical Education

Chen, Belinda Y. MD; Kern, David E. MD, MPH; Kearns, Robert M. MSEd; Thomas, Patricia A. MD; Hughes, Mark T. MD, MA; Tackett, Sean MD, MPH

Author Information
doi: 10.1097/ACM.0000000000002580

Abstract

Online curricula are planned educational experiences with significant online interactions. Online curricula can span institutional, geographic, and temporal boundaries; increase teaching and learning efficiency; and constitute educational scholarship. Versions of online learning have been adopted by most health professions degree and training programs.1–3

Many faculty participate in online curriculum development (CD) to fulfill their educational responsibilities. Although often held accountable for curricular outcomes, faculty may be unprepared to engage in online CD. This can result in faculty frustration, inefficient resource use, and ineffective curricula.2,4–6

Health professions curricula demand sound CD processes because of their responsibility to meet the health care needs of the public. Online CD is particularly accountable to these demands because online curricula can reach large audiences. We describe how a widely accepted, systematic approach to CD for health care education7,8—the Six-Step Approach for Curriculum Development for Medical Education—can be applied to online CD.

In this article we describe and compare four currently popular online curricular formats whose differences have practical implications for online CD (Table 1):

T1
Table 1:
Comparisons of Four Common Formats of Online Curriculaa
  • Blended curricula: Online and face-to-face sessions are combined; the “flipped classroom” is a version of blended learning.
  • Instructor-led fully online curricula: All curricular content is online, and online interaction occurs between faculty and a defined group of learners.
  • Self-paced modules: The curriculum is initiated and directed by the learner, with no additional faculty input during the learner’s experience.
  • Massive open online courses (MOOCs): The curriculum aims for large-scale interactive participation and is open to anyone with Internet access.

Standardized terminologies have also been created for the instructional methods, assessment methods, and resource types used within medical curricula.9

Overview

The Six-Step Approach defines curriculum as any “planned educational experience” and can be applied to short educational sessions or multiyear programs. Though presented sequentially, CD is a continuous, cyclical process, and all steps influence each other (Figure 1). One might begin by considering a specific educational method (Step 4), but must also consider resource and administrative requirements for implementation (Step 5) and the method’s congruency with targeted learners’ needs (Step 2), as well as educational objectives (Step 3) and both learner and program evaluation plans (Step 6). Step 1 is at the top of Figure 1 to emphasize that all health professions CD aims to address a health care problem. Below, we summarize the fundamentals of each step and describe special considerations for each step in online CD.

F1
Figure 1:
Considerations for online curriculum development according to the Six-Step Approach for Curriculum Development for Medical Education.

Step 1: Problem Identification and General Needs Assessment

General principles

In Step 1, the educator identifies the health care problem that the educational intervention will address and, on a regional to international level, characterizes its importance (e.g., prevalence, morbidity, cost). This is typically done by a review of available data (e.g., published literature, public health statistics) and collection of new data (e.g., expert consultation).

The general needs assessment defines the gap between the ideal and current approaches to the identified problem by patients, society, health care educators, and health care professionals. Step 1 may be conducted on a large scale10 or by smaller CD teams. By the end of Step 1, educators can articulate how planned educational activities will prepare health professions trainees to meet the needs of future patients.

Special considerations

Often, Step 1 for online CD is similar to traditional CD, although more effort is required in online CD to search for existing online curricula. Searching online peer-reviewed curriculum repositories (e.g., www.mededportal.org) and reviewing websites of institutions and professional and specialty societies (e.g., www.POGOe.org)8 can yield results. Networking with educators or consulting with medical librarians can help identify other curricula that aim to solve similar problems.

Here is an illustration of how Step 1 influenced online CD to improve tuberculosis (TB) care. TB was identified as a leading cause of morbidity and mortality in low- and middle-income countries (LMICs) despite the existence of effective treatment options. Provider knowledge gaps regarding case-finding and treatment recommendations were found to be an important barrier, making TB care amenable to an educational intervention. Existing curricula were face-to-face, which limited their reach because of travel costs and scheduling logistics. To address these shortcomings, a MOOC titled “Global Tuberculosis Clinical Management and Research” was developed and marketed to providers in LMICs, successfully reaching 7,800 learners, 87% of whom were in settings with high TB prevalence.11

Simply making a curriculum available online may not address health needs. General MOOC data indicate that many may not reach those who need them most.12,13 Clarifying one’s curricular purpose helps the curriculum developer make decisions to maximize educational impact.

Step 2: Targeted Needs Assessment

General principles

Step 2 focuses on the needs of one’s targeted learners and their learning environments. The targeted needs assessment involves the collection of data, such as the characteristics, needs, and preferences of the targeted learners and stakeholders, and available resources in the learners’ environments.

Special considerations

Targeted needs assessments for online CD can vary in scope and complexity. “Flipping the classroom”14 may be relatively simple because the instructor knows the audience. Self-paced modules or MOOCs can be complex because they target diverse groups of geographically dispersed learners.

Online curricula can have high dropout rates related to low social engagement, low perceived relevance, and inadequate self-regulatory behaviors,15,16 so understanding learner needs, preferences, motivations, and past positive and negative experiences with online learning17 is important for maintaining engagement. Learners’ language skills, cultures, preferred modes of accessing content (e.g., laptop, tablet, mobile device), time available for the learning experience, availability for synchronous sessions (e.g., practicing providers to participate in extension for community health outcomes curricula18), the need for credit (e.g., continuing medical education [CME], maintenance of certification), and the willingness to pay for access or recognition of completion may also affect the uptake of an online curriculum.

Environmental factors to evaluate include the availability and quality of Internet access and local technical support. For international curricula, laws and regulations, such as restricted access to media-sharing (e.g., YouTube) and social media (e.g., Facebook) platforms, can affect which sites are chosen for curricular content delivery or communication.

When learners and their environments cannot be fully assessed in advance, parts of a targeted needs assessment can be embedded in online curricula. For example, intentionally gathering feedback from learners during the curriculum can identify confusing terminology or culturally sensitive images needing revision.

Step 3: Goals and Objectives

General principles

Based on the needs assessments, goals and objectives are defined in Step 3. Goals provide direction and set curricular boundaries. Objectives describe specific and measurable expectations for learners’ cognitive, affective, and psychomotor achievements. Objectives can also articulate the anticipated impact on health care. Goals and objectives guide the selection of educational and evaluation strategies in later steps.

Special considerations

Faculty may be tempted to try out the “bells and whistles” of new educational technologies. However, writing objectives based on the health care need delineated in Step 1 and the targeted needs assessment in Step 2 can direct more thoughtful use of established and emerging technologies. Blended curricula can help achieve higher-order objectives in communication and procedural skills by integrating practice and feedback during live sessions. MOOCs and self-paced modules may convey information to a larger number of learners, but are currently limited in their ability to develop learners’ psychomotor skills or change learners’ behaviors. In the future, new technologies such as simulated case scenarios with complex branching logic and virtual environments may allow learners to practice higher-order cognitive thinking, procedures, and even teamwork and communication skills. Generally, one should aim for higher-order objectives19,20 and select technologies accordingly. Also, when technology permits, learners should be engaged in setting their own objectives.21

Step 4: Educational Strategies

General principles

Step 4 involves selecting feasible educational strategies and content to facilitate achievement of educational objectives. Using multiple educational methods congruent with educational objectives reinforces learning and addresses differences in learner preferences.

Special considerations

Online curricula can incorporate multimedia, interactivity, and virtual social components to allow for more flexible, accessible, and individualized learning than traditional classroom-based models. Although there is no formula for optimal online learning,22,23 evidence and principles exist to guide selection of educational strategies. For example, studies have suggested that practice exercises, spaced repetition, feedback, and a social presence are most effective for achieving higher-order learner objectives.6,24 Guidelines for content layout and incorporating multimedia25,26 endorse “chunking” information, combining graphics and words, and simultaneously stimulating auditory and visual pathways. When studies and expert guidelines are insufficient, general educational science and theory can guide educational strategies.

Most online curricula are delivered through learning management systems (LMSs), software that stores and delivers curricular content and facilitates assessment and evaluation. Some LMSs are open-access, although many institutions have already contracted with LMS vendors. LMSs can offer technical support but may also constrain curricular options. New technologies make it possible to access “online curricula” despite poor Internet access,27 and how these interface with LMSs may further influence educational strategies. Knowing characteristics of one’s LMS options is important for LMS selection and instructional design decisions within the curriculum.

Converting a traditional curriculum to an online format can allow faculty to reach a larger audience and overcome classroom space limitations while maintaining synchronous interactions between faculty and learners. Adapting traditional face-to-face methods can be made straightforward by using online synchronous sessions, such as fully online curricula with live sessions that use web conferencing software. Use of special online media rooms can simulate traditional classrooms as an instructor can view learners in real time, including observing facial expressions and body language. Adapting traditional methods to asynchronous online formats, often to increase learner flexibility, is less straightforward, although analogous methods exist (Table 2) and are compatible with most LMSs.

T2
Table 2:
Converting Traditional Educational Methods to Online Asynchronous Modalitiesa

Principles of copyright and fair use are major considerations when choosing the content for online curricula. Copyright is the legal right of a creator of original content to authorize its distribution. Fair use applies to limited use of a copyrighted work for educational purposes. However, when material is widely distributed online or there is a charge for online curricula, the principle of fair use may no longer apply, and legal consultation may be needed. Copyright laws vary internationally, and what is considered fair use is subject to interpretation.28 To avoid violating copyright regulations, consider using publicly available content or content with a Creative Commons license (which permits sharing of an original work with appropriate author attribution).29

Finally, budgets influence choice of methods. Text and ready-made graphics are typically inexpensive, whereas custom graphics and professional videos can cost hundreds or thousands of dollars to produce. We recommend first identifying multiple strategies congruent with one’s objectives, and then selecting based on available resources.

Step 5: Implementation

General principles

Step 5 involves the identification, procurement, and appropriate use of resources (e.g., personnel, time, space and materials, funding), political support, and administrative mechanisms to implement the curriculum. Piloting provides an opportunity to revise a curriculum, based on learner experience, prior to full implementation. When the curriculum is innovative or learners are skeptical, piloting on a friendly audience is wise. Complicated curricula may be phased in as separate segments to focus resources, maximize the likelihood of initial success, and create demand for more.

Special considerations

Implementation for online curricula is often more complex than for traditional curricula30 because of greater personnel and financial requirements; challenges of disseminating to large, diverse audiences; and the need to familiarize both learners and participating faculty with online instruction. Although academic centers vary in how they invest in technology,31 aligning online CD with institutional priorities can provide access to essential resources to manage complexities and streamline CD.

Providing instructions and demonstrations for learners and faculty is often necessary even when navigating online curricula seems straightforward. Some institutions have prerequisite online curricula that introduce LMS navigation. Online curricula can also include brief instructions for navigation at the outset. Ongoing technical support to troubleshoot issues during the curriculum is often necessary. If the LMS vendor does not provide technical support or if multiple systems are used, the institution or sponsoring organization will need to provide technical assistance to avoid diverting significant faculty resources to overcoming technical issues.

Although faculty can independently incorporate online elements when developing blended curricula, one should generally seek to work with team members familiar with different aspects of online CD (Table 3). Faculty usually serve as subject matter experts (SMEs) in collaboration with instructional designers, project managers, and/or media technicians (e.g., videographers, graphic designers). Supplemental Digital Appendix 1, available at https://links.lww.com/ACADMED/A625, provides a guide to common terminology that can be used by online CD teams.

T3
Table 3:
Titles and Roles of an Educational Multidisciplinary Team for Online Curriculaa

Upfront time for resource creation and curricular design is often substantial. Time demanded from SMEs depends on the complexity of the planned curriculum and the availability, experience, and skills of other CD team members.32 Delgaty proposes an estimate of 7 hours of faculty time for each 1 hour of curricular activity4 based on her experience developing an instructor-led fully online curriculum.33 Activities such as moderating discussion boards during the curriculum can be time-consuming; Delgaty4 estimates that 4 hours of faculty time per hour of student activity was necessary. If faculty are unlikely to be available for regular cycles of the curriculum, self-paced curricula may be a prudent choice. Such curricula require no ongoing faculty input after creation, but require much more upfront faculty time. To develop a 3-hour series of self-paced modules in a new LMS without a project manager, our team of 3 faculty each devoted more than 25 hours of planning and feedback for each hour of curricular activity. Having a sense of the time available to SMEs and others on the development team allows one to make appropriate selections among online curricular options and ensure sustainability (Table 1).

The open nature of online curricula requires attention to regulations governing confidentiality and accessibility. U.S. medical education examples include the Health Insurance Portability and Accountability Act (HIPAA), which requires that personal health information be protected; the Family Educational Rights and Privacy Act (FERPA), which requires that student information be kept confidential; and the Americans with Disabilities Act (ADA), which requires that individuals with disabilities (including, but not limited to, hearing or vision impairment) have access to the curriculum. Because most institutional LMSs already ensure that confidentiality requirements are met, the CD team can usually focus on accessibility requirements by captioning video content and providing “alt tags” (wording that appears in place of images for the visually impaired).

Online curricular budgets range considerably (Table 1) and should include the cost of development, implementation, and maintenance. Some faculty scale back goals and objectives to fit available resources, while others may successfully garner additional resources based on the strength of the problem identification and needs demonstrated during Steps 1 and 2.

Online curricula can be free and open-access, require payment for access, or adopt a hybrid approach (e.g., anyone may access, but receiving educational credit requires payment). Online curricula are the most-used type of CME activity, which generated $1.3 billion in registration fees in 2016.34 MOOCs often charge $25 to $50 for a certificate; popular MOOCs can generate significant returns as direct revenue. Online curricula that are not independently profitable may be valuable as advertising for enrollment in complementary academic programs.

Charging fees can support curriculum revision and maintenance, but may limit the uptake and impact of an online curriculum. If planning to charge fees, one must consider additional factors: How will the online curriculum integrate with a payment portal? Will fees be paid by individuals or institutions? Will fees be based on individual curricular components or on a time-limited basis? How will revenue be distributed among curriculum faculty, their divisions or departments, and the institution?

Faculty seeking to disseminate curricula broadly must also consider issues of branding—the policies governing use of an organization’s name or logo—and may need to consider intellectual property ownership. Curriculum developers may also wish to protect parts of their curricula from undesired use, alteration, or dissemination beyond their control and, where concerns exist, should seek legal advice.

Step 6: Evaluation and Feedback

General principles

Step 6 evaluates whether the curriculum was successful and how it can be improved. It includes 10 tasks: identifying evaluation users, articulating user needs, prioritizing evaluation questions, choosing evaluation designs (e.g., posttest only, pre/posttest, comparison group), selecting evaluation methods, constructing instruments, addressing ethical concerns (e.g., confidentiality), collecting data, analyzing data, and reporting results. Step 6 includes assessment of individual participants and evaluation of the program’s structures, processes, and outcomes. Qualitative and quantitative data should be collected for formative and summative purposes. Evaluation methodology must be feasible and congruent with educational objectives.

Special considerations

Blended curricula offer the same assessment options as traditional curricula because they include face-to-face sessions. Even in fully online curricula, instructors can review uploaded documents and multimedia files through an LMS and track participation in online discussions. Self-paced modules and MOOCs have the fewest assessment options as both typically require automated grading. MOOCs also offer peer grading, although this must be thoughtfully designed to ensure validity and enhance learner engagement; pairing enrollees of similar backgrounds and calibrating and weighting the quality of peer assessments against the instructor’s may be beneficial.35

For online formative assessments, mobile devices can allow assessments to occur at any time and in any location. Learning dashboards can show learners how their responses compare to others in real time. Gamification features can help learners increase effort and focus to enhance learning.36,37 For online summative assessments, safeguards for exam security and integrity are often necessary.

Evaluation of online curricula should generally include a more robust evaluation of process/implementation, assessing nontraditional factors such as usability, reusability, sustainability, learner experience, and satisfaction with the technology.38 Online curricula also offer the opportunity to conduct “A/B testing,” where two alternatives are compared with rapid evaluation and adjustment. Because data often must be collected electronically during learners’ participation in the curriculum, certain evaluation designs might be challenging, such as those using a comparison group (which might be difficult to identify and access) or those requiring longitudinal follow-up (where response rates might be low because learners are dispersed).

Online curricular assessments and evaluations provide new opportunities to advance the science of teaching and learning. For example, most platforms automatically track variables such as the locations of learners and time spent engaging in the curriculum. The relatively new field of learning analytics uses data from online learning to create predictive models and guide learners in real time. Assessments can be automatically collated for large numbers of learners to track performance across cohorts and over time. Harnessing the “big data”39 generated by combining location and time information with frequent and repeated assessments has the potential to provide greater granularity for how learning occurs.40,41

Maintenance and Enhancement

General principles

No curriculum is perfect. After full curriculum implementation, all elements of the Six Steps should be revisited to ensure that necessary resources and support remain secure and that the curriculum undergoes continuous renewal. Curricular content and design should remain up-to-date and aligned with important health care needs. One needs to establish processes for prioritizing and managing revisions, given resource constraints.

Special considerations

In online curricula, in addition to maintaining financial support for the curricular team, it is often necessary to pay annual LMS fees and fund ongoing technical support.

Because many online curricula do not conform to a traditional academic calendar, the revision of curricular content and methods will depend on timing, type of content, learner expectations, availability of the online CD team, and budget. Keeping content and educational methods on the cutting edge must be balanced with the resources required to implement such changes.

Online Curricula as Scholarship

General principles

Although some institutions have recognized changing definitions of scholarship,39,42,43 most endorse Glassick’s44 well-accepted criteria, which can be satisfied by proceeding through the Six Steps (with each step offering opportunities for scholarship).45 Steps 1 and 2 ensure adequate preparation and clear goals, Step 3 articulates those goals and leads to appropriate methods (Steps 4 and 5), and Step 6 yields significant results and reflective critique. Where most traditional curricula fall short of becoming true scholarship is in the realm of effective presentation, which requires dissemination and availability for peer review.

Special considerations

Dissemination can occur by the spread of the actual curriculum beyond one’s home institution or by presentation or publication of curriculum-related work.4 Some curriculum repositories, such as MedEdPORTAL, require peer review before materials are placed online. Online curricula such as MOOCs or self-paced modules are by definition “disseminated” as they are made available to large audiences; peer review of these curricula can occur through their evaluations. Sharing evaluations from online curricula can also provide regional, national, or international information on the knowledge or skills of targeted groups.46,47

Blended and instructor-led curricula often contain reusable learning objects (RLOs)—components such as images, case studies, diagrams, or assessments that can be shared and reused for other settings. RLOs that have undergone rigorous development and evaluation can qualify as scholarship.

Health professions faculty can also pursue opportunities to engage in educational scholarship around online learning. Information generated from applying a methodical CD process advances our understanding of how to both standardize and individualize learning. Sharing curricula and curricular components will reduce duplication of effort, expand the number of learners influenced by each work, and promote professional advancement.

Concluding Observations

Online approaches to health care education can reduce duplicative CD efforts, increase the quality of curricula, and document achievement by a large number of learners. They can be, to varying degrees, asynchronous and self-paced, and potentially more personalized and time-efficient for learners and educators. Although they may require a relatively large upfront investment of resources, their instructional and assessment advantages, their reach across geographical and institutional boundaries, and their potential to affect health care outcomes can justify the effort. Applying the Six-Step Approach with an understanding of factors particular to online CD can lead to more effective and scholarly educational program development.

References

1. Chumley-Jones HS, Dobbie A, Alford CL. Web-based learning: Sound educational method or hype? A review of the evaluation literature. Acad Med. 2002;77(10 suppl):S86–S93.
2. Cook DA, Garside S, Levinson AJ, Dupras DM, Montori VM. What do we mean by web-based learning? A systematic review of the variability of interventions. Med Educ. 2010;44:765–774.
3. Kamin C, Souza KH, Heestand D, Moses A, O’Sullivan P. Educational technology infrastructure and services in North American medical schools. Acad Med. 2006;81:632–637.
4. Delgaty L. Twelve tips for academic role and institutional change in distance learning. Med Teach. 2015;37:41–46.
5. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions: A meta-analysis. JAMA. 2008;300:1181–1196.
6. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Instructional design variations in Internet-based learning for health professions education: A systematic review and meta-analysis. Acad Med. 2010;85:909–922.
7. Thomas PA, Kern DE, Hughes MT, Chen BY. Curriculum Development for Medical Education: A Six-Step Approach. 2016.3rd ed. Baltimore, MD: Johns Hopkins University Press.
8. Thomas PA, Kern DE. Internet resources for curriculum development in medical education: An annotated bibliography. J Gen Intern Med. 2004;19(5 pt 2):599–605.
9. Association of American Medical Colleges. MedBiquitous Curriculum Inventory Working Group Standartized Vocabulary Subcommittee. Curriculum inventory. Standardized instruction and assessment methods and resource types. https://medbiq.org/curriculum/vocabularies.pdf. Published March 2016. Accessed November 21, 2018.
10. Cooke M, Irby DM, O’Brien BC. Educating Physicians: A Call for Reform of Medical School and Residency. 2010.Hoboken, NJ: John Wiley & Sons.
11. Johns Hopkins School of Nursing. Global tuberculosis clinical management and research. http://learn.nursing.jhu.edu/online/mooc/tb.html. Accessed November 21, 2018.
12. Hansen JD, Reich J. Democratizing education? Examining access and usage patterns in massive open online courses. Science. 2015;350:1245–1248.
13. Emanuel EJ. MOOCs taken by educated few. Nature. 2013;503:342.
14. Moffett J. Twelve tips for “flipping” the classroom. Med Teach. 2015;37:331–336.
15. Park JH, Choi HJ. Factors influencing adult learners’ decision to drop out or persist in online learning. J Educ Technol Soc. 2009;12:207–217.
16. Nawrot I, Doucet A. Building engagement for MOOC students: Introducing support for time management on online learning platforms. Paper presented at: 23rd International World Wide Web Conference, Workshop on Web-Based Education Technologies; April 2014; Seoul, South Korea. https://hal.archives-ouvertes.fr/hal-01075255/file/ACTI-NAWROT-2014-2.pdf. Accessed November 21, 2018.
17. Cook DA, Thompson WG. Comfort and experience with online learning: Trends over nine years and associations with knowledge. BMC Med Educ. 2014;14:128.
18. University of New Mexico. Project ECHO: A revolution in medical education and care delivery. https://echo.unm.edu/about-echo/model. Accessed November 21, 2018.
19. Anderson LW, Krathwohl DR. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. 2001.New York, NY: Addison Wesley Longman.
20. Engelhart MD, Furst EJ, Hill WH, Krathwohl DR. Bloom BS. Taxonomy of Educational Objectives: Handbook I: Cognitive Domain. 1956.New York, NY: Addison Wesley Longman.
21. Lau KH. Computer-based teaching module design: Principles derived from learning theories. Med Educ. 2014;48:247–254.
22. Rowe M, Frantz J, Bozalek V. The role of blended learning in the clinical education of healthcare students: A systematic review. Med Teach. 2012;34:e216–e221.
23. Onyura B, Baker L, Cameron B, Friesen F, Leslie K. Evidence for curricular and instructional design approaches in undergraduate medical education: An umbrella review. Med Teach. 2016;38:150–161.
24. Cook DA, Triola MM. What is the role of e-learning? Looking past the hype. Med Educ. 2014;48:930–937.
25. Yavner SD, Pusic MV, Kalet AL, et al. Twelve tips for improving the effectiveness of web-based multimedia instruction for clinical learners. Med Teach. 2015;37:239–244.
26. Mayer R. The Cambridge Handbook of Multimedia Learning. 2014.Cambridge, UK: Cambridge University Press.
27. Learning Equality. https://learningequality.org. Accessed November 21, 2018.
28. U.S. Copyright Office. More information on fair use. https://www.copyright.gov/fair-use/more-info.html. Accessed November 21, 2018.
29. Creative Commons. Licenses. https://creativecommons.org/licenses. Accessed November 21, 2018.
30. Cook DA. The value of online learning and MRI: Finding a niche for expensive technologies. Med Teach. 2014;36:965–972.
31. Association of American Medical Colleges. Summary of select results from the medical school information technology survey sponsored by the Group on Information Resources (GIR). https://www.aamc.org/download/430608/data/2014girmedicalschoolitsurveysummary.pdf. Published April 2015. Accessed November 21, 2018.
32. Chapman Alliance. LMS selection services, learning technology consulting, choosing an LMS. How long does it take to create learning? http://www.chapmanalliance.com/howlong. Accessed November 21, 2018.
33. Delgaty L. A critical examination of the time and workload involved in the design and delivery of an e-module in postgraduate clinical education. Med Teach. 2013;35:e1173–e1180.
34. 2017 Accreditation Council for Continuing Medical Education. ACCME data report. Table 6. http://www.accme.org/sites/default/files/2018-04/754_20170712_2016_Data_Report_3.pdf. Accessed November 21, 2018.
35. Suen HK. Peer assessment for massive open online courses (MOOCs). Int Rev Res Open Distrib Learn. July 2014;15. http://www.irrodl.org/index.php/irrodl/article/view/1680/2904. Accessed November 21, 2018.
36. Pitt MB, Borman-Shoap EC, Eppich WJ. Twelve tips for maximizing the effectiveness of game-based learning. Med Teach. 2015;37:1013–1017.
37. Evans KH, Daines W, Tsui J, Strehlow M, Maggio P, Shieh L. Septris: A novel, mobile, online, simulation game that improves sepsis recognition and management. Acad Med. 2015;90:180–184.
38. Cook DA, Ellaway RH. Evaluating technology-enhanced learning: A comprehensive framework. Med Teach. 2015;37:961–970.
39. Ellaway R. Scholarship in an age of big data. Med Teach. 2013;35:613–615.
40. Pusic MV, Boutis K, Hatala R, Cook DA. Learning curves in health professions education. Acad Med. 2015;90:1034–1042.
41. Haynes MR, Gaglani S, Wilcox M, Mitchell T, DeLeon V, Goldberg H. Learning through Osmosis: A collaborative platform for medical education. Innov Glob Med Health Educ. 2014;2:1–8. http://www.qscience.com/doi/full/10.5339/igmhe.2014.2. Accessed January 23, 2019.
42. Thoma B, Chan TM, Benitez J, Lin M. Educational scholarship in the digital age: A review and analysis of scholarly products. The Winnower. 2014;1:e141827.77297.
43. Sherbino J, Arora VM, Van Melle E, Rogers R, Frank JR, Holmboe ES. Criteria for social media-based scholarship in health professions education. Postgrad Med J. 2015;91:551–555.
44. Glassick CE. Boyer’s expanded definitions of scholarship, the standards for assessing scholarship, and the elusiveness of the scholarship of teaching. Acad Med. 2000;75:877–880.
45. Kern D, Bass E. Thomas PA, Kern DE, Hughes MT, Chen BY. Dissemination. In: Curriculum Development for Medical Education: A Six-Step Approach. 2016:3rd ed. Baltimore, MD: Johns Hopkins University Press; 194–195.
46. Sisson SD, Hughes MT, Levine D, Brancati FL. Effect of an Internet-based curriculum on postgraduate education. A multicenter intervention. J Gen Intern Med. 2004;19(5 pt 2):505–509.
47. Sisson SD, Dalal D. Internal medicine residency training on topics in ambulatory care: A status report. Am J Med. 2011;124:86–90.
48. Puzziferro M, Shelton K. A model for developing high-quality online courses: Integrating a systems approach with learning theory. J Asynchronous Learn Netw. 2008;12:119–136.

    Supplemental Digital Content

    Copyright © 2019 by the Association of American Medical Colleges