Pediatric Resident Engagement With an Online Critical Care Curriculum During the Intensive Care Rotation* : Pediatric Critical Care Medicine

Journal Logo


Pediatric Resident Engagement With an Online Critical Care Curriculum During the Intensive Care Rotation*

Daniel, Dennis A. MD1,2; Poynter, Sue E. MD, MEd3; Landrigan, Christopher P. MD, MPH4–6; Czeisler, Charles A. PhD, MD6,7; Burns, Jeffrey P. MD, MPH1,2; Wolbrink, Traci A. MD, MPH1,2

Author Information
Pediatric Critical Care Medicine 21(11):p 986-991, November 2020. | DOI: 10.1097/PCC.0000000000002477


In busy clinical rotations such as the ICU, patients, diagnoses, and clinical acuity vary between rotations, and limited time and competing demands are common for both trainees and faculty (1). Supplementing the ICU rotation experience with a blended online educational curriculum may help improve knowledge and ensure consistent exposure to core content (2), but these resources are often superimposed on the demands of patient care and other educational experiences (3). For our ICU residents, we designed curricula that included short videos with pre- and posttests and hypothesized that residents would use these materials most frequently during breaks in patient care while on clinical duty. However, there is a lack of previously published data that describe when and to what extent resident physicians complete online materials during ICU rotations. Such understanding would help inform decisions about how to best implement educational interventions for residents rotating in the demanding, high-acuity clinical setting of the ICU.


We conducted a prospective cohort study examining junior (post-graduate year 2) resident use of online curricula during their first rotation in medical-surgical PICUs at two large academic children’s hospitals (Boston Children’s Hospital and Cincinnati Children’s Hospital) that are similar in size, patient population, and resident scope of responsibility. This study was approved by the Institutional Review Boards at both sites. ICU resident rotation directors created individualized curricula for each site that covered core concepts in pediatric critical care medicine. Both sites also provided in-person educational experiences (bedside teaching, didactic lectures, and manikin-based simulations). Site 1 delivered in-person education between 07:30 and 08:15 every weekday, and Site 2 did so between 12:15 and 13:00 every Monday through Thursday. We administered the online curricula from July 2013 to June 2015 at Site 1 and June 2015 to June 2017 at Site 2. Neither site provided protected time for curriculum completion. Although residents were informed of the expectation to complete curricula by the end of their ICU rotation, there were no formal consequences for failing to complete. At both sites, residents were e-mailed 2 weeks before their rotation and instructed to complete the curriculum before the end of their ICU rotation, with e-mail reminders provided at rotation weeks 2, 3, and 4. Residents were excluded from the time-of-use analysis if their daily schedule data were unavailable.

Curricula were delivered on OPENPediatrics (, which is based at Site 1. Each lesson contained a pretest, video, and posttest. The curriculum contained 18 lessons at Site 1 and 21 lessons at Site 2. Individual lesson videos ranged in length from 4 to 36 minutes (average length 18 min). The total curriculum video duration at Site 1 was 5 hours, 12 minutes and at Site 2 was 6 hours, 50 minutes. The website required strictly linear progress; a pretest, then video, then posttest for each lesson needed to be completed before a resident could progress to the next lesson. Residents were only able to take the pretest once, but posttests could be attempted multiple times until the minimum passing score (≥80%) was achieved. The platform allowed residents to stop and restart within pre- or posttests, as well as within videos, if they did not complete a given item in one sitting.

During the study interval, residents worked within two different clinical schedules as part of a concurrent trial of resident physician work hours Randomized Order Safety Trial Evaluating Resident Schedules (4), where each schedule operated for 1 year of clinical rotations at each site. One schedule cohort involved daytime and nighttime work shifts limited to a maximum of 16 hours of duration, whereas the second involved traditional extended duration (>24 hr) work shifts, with daytime shifts alternating with extended duration work shifts every fourth night. On average, residents worked about 10% more hours per week on the extended duration work schedule (5). Throughout this article, we refer to these differing schedule cohorts as “short call” and “long call”.

We collected curriculum completion data for each resident and timestamps for every pre- and the first posttest attempt that occurred during the ICU rotation and in the 14 days preceding. We did not include test attempts occurring prior to the ICU rotation in the time-of-use analysis due to the significant heterogeneity in resident clinical rotations immediately prior to the ICU rotation. We only included the first posttest attempt to avoid over-representing a given time of day if a user attempted a posttest multiple times. Video viewing activity is captured only in aggregate, deidentified fashion on the platform, so individual video view timestamps were not available for specific residents. For test attempts during the ICU rotation, we cross-referenced timestamps with each resident’s schedule to determine whether or not they were scheduled for clinical service in the hospital. We calculated frequencies and percentages for resident and site characteristics and compared data between cohorts and between sites using chi-square tests of independence using a significance level of 0.05. Data were analyzed using Stata/SE 13.1 (StataCorp, College Station, TX) and Microsoft Excel (Microsoft Corp., Washington, DC).


During the study, 157 residents rotated through the ICU for the first time, and 92% (144/157) accessed the curriculum. Fifty-three percent of residents (83/157) accessed the curriculum during the ICU rotation at least once while on duty. Seven residents at Site 2 accessed the curriculum during the rotation but did not have daily schedule data available for analysis. We included 137 residents (137/157; 87%) in the time-of-use analysis. There were no statistically significant differences in demographic characteristics (gender or residency track) between the two sites or between the schedule cohorts within each site.

Curriculum Completion

Sixty-eight percent of residents (106/157) completed the curriculum (Table 1). A greater percentage of residents completed the curriculum at Site 2 (53/60; 88%) compared with Site 1 (53/84; 63%), p value of less than 0.001. There was no statistically significant association between overall curriculum completion and schedule cohort at either site.

TABLE 1. - Resident Learning Tracks and Curriculum Completion Rates by Cohort and Site
Study Participant Category Site 1 Site 2 Overall
Short Call, n (%) Long Call, n (%) p Short Call, n (%) Long Call, n (%) p Overall, n (%) p (between sites)
Total residents, n 48 49 32 28 157
Residency track
 Categorical 40 (83) 42 (86) 0.75 23 (73) 21 (75) 0.79 126 (80) 0.09
 Medicine-Pediatrics 4 (9) 4 (8) 0.98 3 (9) 2 (7) 0.76 13 (8) 0.98
 Pediatric-Neurology 3 (6) 2 (4) 0.63 3 (9) 3 (11) 0.86 11 (7) 0.25
 Other combined tracksa 1 (2) 1 (2) 0.99 3 (9) 2 (7) 0.76 7 (5) 0.06
Curriculum access and completion
 Accessed curriculum 41 (85) 43 (88) 0.73 32 (100) 28 (100) 144 (92) 0.003
 Completed curriculum overall 29 (60) 24 (49) 0.26 30 (94) 23 (82) 0.16 106 (68) < 0.001
 Missing schedule data 0 0 0 7 (25) 0.003 7 (4) 0.001
 Included in time-of-use analysis 41 (85) 43 (88) 0.74 32 (100) 21 (75) 0.003 137 (87) 0.75
aPediatrics-Anesthesia, Pediatrics-Genetics, Pediatrics-Physical Medicine and Rehabilitation, Pediatrics-Psychiatry-Child Psychiatry.
Dashes indicate a test of significance was not run for these items.

Time-of-Use Analysis

We included 2,828 test attempts by 137 residents from the two sites in the time-of-use analysis. Of the test attempts made during clinical duty, 40% (418/1,043) occurred during daytime shifts versus 60% (625/1,043) at night. Approximately two thirds of test attempts (1,785/2,828; 63%) occurred when residents were not scheduled to work, regardless of time of day. Approximately two thirds of all test attempts (1,824/2,828; 64%) occurred during nighttime hours (between 6 pm and 6 am), regardless of work status (Fig. 1A) (Supplemental Table 1, Supplemental Digital Content 1,

Figure 1.:
Time of use analyses. A, Distribution of test attempts during clinical rotation by hour of day and by on-duty versus off-duty time status. n = 2,828 test attempts. B, Distribution of test attempts in relation to day of rotation (range day –14 to +31). n = 3,176 test attempts (includes 348 attempts prior to ICU rotation).

Forty-two percent of all test attempts (1,199/2,828) occurred during nighttime hours when the residents were not scheduled to work. Nearly a quarter of all test attempts (647/2,828; 23%) occurred between midnight and 6 am, with 12% (342/2,828) of all test attempts occurring between midnight and 6 am while residents were not scheduled to work.

We observed an association between time-of-use patterns and schedule cohort at Site 1 but not at Site 2. At Site 1, residents in the long call cohort used the curriculum more during nighttime hours (428/553; 77%) compared with residents in the short call cohort (427/769; 56%), p value of less than 0.001. The long call cohort also used the curriculum more during times when they were not scheduled to work (461/553; 83%) compared with the short call cohort (512/769; 67%), p value of less than 0.001.

Figure 1B displays the distribution of test attempts by day of rotation, ranging from 14 days prior to the start of the rotation to 31 days after the start date. For test attempts within the ICU rotation, a greater proportion occurred in the second half of the rotation (1,679/2,828; 60%) versus the first half (1,149/2,828; 40%). Three hundred and forty-eight additional test attempts occurred prior to the start of the rotation, the majority of which (264/348; 76%) were in the 2 days immediately prior.


Using 4 years of timestamped online learning data from two large pediatric residency programs, we found that the majority of residents in our study accessed a supplemental online curriculum during the ICU rotation. However, despite designing the ICU curriculum to include short lessons that could be completed during breaks from clinical work while on duty, only half of the included residents accessed the curriculum during clinical periods, and they frequently chose to use the materials during nighttime hours and when not scheduled to work clinically. Notably, 12% of test attempts occurred between midnight and 6 am while residents were not scheduled to work. Test attempts were made immediately prior to, and throughout the rotation, with a majority occurring in the second half of the rotation.

The inconsistent association of completion rate or time of use with schedule cohort makes it less likely that the type of schedule is a main contributor to online curriculum engagement. Because both sites provided in-person educational experiences during most workdays in addition to the online curriculum, it is possible that residents were biased against completing online materials while on-duty, since other materials were already being presented during on-duty hours. Other studies have supported the notion that medical learners often prefer online learning to be supplementary to in-person learning experiences (6,7). The greater proportion of on-duty use at night compared with during the daytime may be a consequence of the greater volume of clinical care demands requiring resident attention during the day, including but not limited to rounds and scheduled admissions. The greater number of test attempts in the second half of the rotation may reflect residents catching up on incomplete lessons before the end of the rotation or may be related to residents focusing on getting comfortable in the clinical environment before turning their attention to self-directed learning.

Several studies have highlighted successful implementation and outcomes of online medical learning, noting improvements in knowledge (8) and perceived utility and satisfaction from clinicians and instructors (9). However, although residents have always incorporated self-directed learning at night and during off hours, blended learning models that increase trainees’ obligatory nonclinical workload outside of dedicated educational time may have a different impact. Despite our intention to provide short lessons that would provide education during breaks from patient care, residents accessed the curriculum more often during nonclinical hours. Previous reports have commented on the need to consider the distinct time constraints and service-education task conflicts in graduate medical education (10) and on the risk of creating information overload when educational content is shifted to the online environment (11). Increases in workload added to preexisting stressors of the clinical learning environment can contribute to resident physician burnout (12,13) and sleep deprivation. Sleep deficiency is known to adversely impact resident clinical performance (14) and increases risk of physical harm, such as motor vehicle crashes (15) and needlestick/sharps injuries (16). Therefore, program and rotation directors of busy, inpatient rotations may need to consider alternative approaches to implementing online learning, such as incorporating protected time to complete curricula; making the curriculum optional or controlling the volume of content shifted to the asynchronous, self-directed setting (11,17,18).

Our study has several important limitations. As this was a purely an observational study, we did not qualitatively assess resident motivations for why they accessed the curricula at the times they did nor did we formally survey participant satisfaction with the curriculum or any potential impact on their wellness. These are important areas that warrant further investigation. Additionally, not all residents completed the online curricula, and rates of curriculum completion differed between the two sites despite having identical procedures to encourage completion. This may reflect differences in willingness to engage with online curricula during clinical rotations between residents and between the two sites, which may have led to a sampling bias. Interestingly, despite the fact that OPENPediatrics ( is primarily based at Site 1, curriculum completion rate was lower at that site, suggesting that there was not increased pressure among residents to use the platform due to the shared institutional affiliation. Although we have no reason to suspect differences in resident roles or workload between the sites, we did not directly assess this either. Curricular length has also been described as a barrier to curriculum completion (19,20); however, we observed higher completion rates by residents at the site with the longer curriculum.


Our data show that residents will engage with online learning materials during and immediately prior to their ICU rotation but do so most often at nighttime and when off-duty, with a portion of use occurring during midnight and 6 am while off-duty. This may have implications for resident well-being, including sleep, personal life disruptions, and/or burnout, and warrants further examination. Program and rotation directors will need further guidance on how best to implement blended learning models in busy clinical rotations, such as the ICU.


We wish to thank the residents at Boston Children’s Hospital and Cincinnati Children’s Hospital for their participation in this study, as well as the administrative staff at both hospitals and at OPENPediatrics who provided coordination and support for this study.


1. Turner TL, Fielder E, Ward MA. Balancing service and education in residency training: A logical fallacy. JAMA Pediatr. 2016; 170:101–102
2. Wittich CM, Agrawal A, Wang AT, et al. Flipped classrooms in graduate medical education: A national survey of residency program directors. Acad Med. 2018; 93:471–477
3. Chen F, Lui AM, Martinelli SM. A systematic review of the effectiveness of flipped classrooms in medical education. Med Educ. 2017; 51:585–597
4. Blackwell T, Kriesel DR, Vittinghoff E, et al.; ROSTERS Study Group. Design and recruitment of the randomized order safety trial evaluating resident-physician schedules (ROSTERS) study. Contemp Clin Trials. 2019; 80:22–33
5. Barger LK, Sullivan JP, Blackwell T, et al. Effects on resident work hours, sleep duration, and work experience in a randomized order safety trial evaluating resident-physician schedules (ROSTERS). Sleep. 2019; 42:zsz110
6. Morton CE, Saleh SN, Smith SF, et al. Blended learning: How can we optimise undergraduate student engagement?. BMC Med Educ. 2016; 16:195
7. Means B, Toyama Y, Murphy R, et al. The effectiveness of online and blended learning: A meta-analysis of the empirical literature. Teach Coll Rec. 2013; 115:1–47
8. Cook DA, Levinson AJ, Garside S, et al. Internet-based learning in the health professions: A meta-analysis. JAMA. 2008; 300:1181–1196
9. Ruiz JG, Mintzer MJ, Leipzig RM. The impact of E-learning in medical education. Acad Med. 2006; 81:207–212
10. Cooper AZ, Hsieh G, Kiss JE, et al. Flipping out: Does the flipped classroom learning model work for GME?. J Grad Med Educ. 2017; 9:392–393
11. Moffett J. Twelve tips for “flipping” the classroom. Med Teach. 2015; 37:331–336
12. Dyrbye L, Shanafelt T. A narrative review on burnout experienced by medical students and residents. Med Educ. 2016; 50:132–149
13. Landrigan CP, Rothschild JM, Cronin JW, et al. Effect of reducing interns’ work hours on serious medical errors in intensive care units. N Engl J Med. 2004; 351:1838–1848
14. Barger LK, Cade BE, Ayas NT, et al.; Harvard Work Hours, Health, and Safety Group. Extended work shifts and the risk of motor vehicle crashes among interns. N Engl J Med. 2005; 352:125–134
15. Jennings ML, Slavin SJ. Resident wellness matters: Optimizing resident education and wellness through the learning environment. Acad Med. 2015; 90:1246–1250
16. Lockley SW, Barger LK, Ayas NT, et al.; Harvard Work Hours, Health and Safety Group. Effects of health care provider work hours and sleep deprivation on safety and performance. Jt Comm J Qual Patient Saf. 2007; 33:7–18
17. Sait MS, Siddiqui Z, Ashraf Y. Advances in medical education and practice: Student perceptions of the flipped classroom. Adv Med Educ Pract. 2017; 8:317–320
18. Kim S, Willett LR, Pan WJ, et al. Impact of required versus self-directed use of virtual patient cases on clerkship performance: A mixed-methods study. Acad Med. 2018; 93:742–749
19. Wray A, Bennett K, Boysen-Osborn M, et al. Efficacy of an asynchronous electronic curriculum in emergency medicine education in the United States. J Educ Eval Health Prof. 2017; 14:29
20. Jordan K. Massive open online course completion rates revisited: Assessment, length and attrition. The International Review of Research in Open and Distributed Learning. 2015; 16:341-358

duty hours; graduate medical education; online learning

Supplemental Digital Content

Copyright © 2020 The Author(s). Published by Wolters Kluwer Health, Inc. on behalf of the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies.