Purpose: To evaluate educational experiences of internal medicine interns before and after maximum shift lengths were decreased from 30 hours to 16 hours.
Method: The authors compared educational experiences of internal medicine interns at Vanderbilt University Medical Center before (2010; 47 interns) and after (2011; 50 interns) duty hours restrictions were implemented in July 2011. The authors compared number of inpatient encounters, breadth of concepts in notes, exposure to five common presenting problems, procedural experience, and attendance at teaching conferences.
Results: Following the duty hours restrictions, interns cared for more unique patients (mean 118 versus 140 patients per intern, P = .005) and wrote more history and physicals (mean 73 versus 88, P = .005). Documentation included more total concepts after the 16-hour maximum shift implementation, with a 14% increase for history and physicals (338 versus 387, P < .001) and a 10% increase for progress notes (316 versus 349, P < .001). There was no difference in the median number of selected procedures performed (6 versus 6, P = 0.94). Attendance was higher at the weekly chief resident conference (60% versus 68% of expected attendees, P < .001) but unchanged at morning report conferences (79% versus 78%, P = .49).
Conclusions: Intern clinical exposure did not decrease after implementation of the 16-hour shift length restriction. In fact, interns saw more patients, produced more detailed notes, and attended more conferences following duty hours restrictions.
Dr. Theobald is instructor of medicine, Department of Medicine, School of Medicine, Vanderbilt University, Nashville, Tennessee, and VA Quality Scholars fellow, VA Tennessee Valley Healthcare System, Nashville, Tennessee.
Dr. Stover is fellow in medical oncology, Dana-Farber Cancer Institute and Massachusetts General Hospital Cancer Center, Boston, Massachusetts.
Dr. Choma is executive medical director for quality and patient safety and assistant professor, Department of Medicine, School of Medicine, Vanderbilt University, Nashville, Tennessee.
Dr. Hathaway is assistant professor, Department of Medicine, School of Medicine, Vanderbilt University, Nashville, Tennessee, and chief of primary care service, VA Tennessee Valley Healthcare System, Nashville, Tennessee.
Dr. Green is codirector of the primary care medicine clerkship and assistant professor, Departments of Medicine and Pediatrics, School of Medicine, Vanderbilt University, Nashville, Tennessee.
Dr. Peterson is associate director of the internal medicine residency program and assistant professor, Department of Medicine, School of Medicine, Vanderbilt University, Nashville, Tennessee.
Dr. Sponsler is assistant professor, Department of Medicine, School of Medicine, Vanderbilt University, Nashville, Tennessee, and staff physician, VA Tennessee Valley Healthcare System, Nashville, Tennessee.
Dr. Vasilevskis is assistant professor, Department of Medicine, School of Medicine, Vanderbilt University, Nashville, Tennessee.
Dr. Kripalani is chief of hospital medicine and associate professor, Department of Medicine, School of Medicine, Vanderbilt University, Nashville, Tennessee.
Dr. Sergent is director, Internal Medicine Residency Program, and professor, Department of Medicine, School of Medicine, Vanderbilt University, Nashville, Tennessee.
Dr. Brown is chair and professor, Department of Medicine, School of Medicine, Vanderbilt University, Nashville, Tennessee.
Dr. Denny is assistant professor, Departments of Medicine and Biomedical Informatics, School of Medicine, Vanderbilt University, Nashville, Tennessee.
The authors have informed the journal that the research team for this report agrees that both Dr. Theobald and Dr. Stover completed the intellectual and other work typical of the work of the first author.
Correspondence should be sent to Dr. Theobald, VA Tennessee Valley Healthcare System, GRECC, 1310 24th Ave. S., Nashville, TN 37212; telephone: (615) 873-8024; fax: (615) 873-7981; e-mail: email@example.com.
Medical resident duty hours have been increasingly scrutinized over the past two decades. Following the implementation of Code 405 by the New York Department of Health in 1989, the Accreditation Council for Graduate Medical Education (ACGME) first placed limitations on resident duty hours in July 2003.1 The Institute of Medicine recommended further restrictions in 2008, citing concern that extended shifts could contribute both to reduced resident well-being and also increased medical errors.2 The ACGME further revised duty hours limits to include a maximum of 16 hours per shift for postgraduate year one residents (i.e., interns), effective July 2011.3 These updated limitations generated substantial debate because they effectively eliminated all overnight calls for interns.
Since the beginning of duty hours reform in the late 1980s, more than 100 studies have evaluated the impact of these progressive restrictions on both patient care and resident outcomes across the spectrum of medical specialties. Several recent systematic reviews demonstrated that these investigations varied widely in methodology but were primarily single-institution, pre–post, or cross-sectional studies.4–6 Whereas many focused on patient safety outcomes and resident or faculty satisfaction, several focused on resident education metrics—a growing concern in a rapidly changing educational environment.
To date, the methods used to assess the effect of previous duty hours restrictions on resident education have included procedural/operative volume,7–13 standardized medical knowledge test ing,7,9,11,14–16 didactic lecture attendance,17,18 and time spent reading.19–23 Many of these studies largely relied on self-reported data, which is subject to recall bias, and manual chart review. Most studies did not demonstrate a significant impact on procedural volume or didactic participation. Standardized test scores were largely unchanged or slightly improved after duty hours limitations, and residents generally reported an improvement in time available for reading. Studies using objective data have demonstrated no difference in the number of patients admitted, mean census,24 standardized medical knowledge test scores,25 or didactic lecture attendance.26,27
Unintended consequences of duty hours changes, particularly the impact on resident education, are a major concern at academic medical centers (i.e., teaching hospitals). Rigorous evaluation of these changes is critical. A 2005 systematic review noted that there are “no studies that measured actual experience of residents in internal medicine, pediatrics, family medicine.”5 In the interim, several studies4,6,9–15 from different specialties evaluated some factors that affect resident education; however, objective evaluation of internal medicine resident education has lagged relative to other specialties.
Before implementation of the 2011 ACGME duty hours restrictions, a survey by the Association of Program Directors in Internal Medicine demonstrated that 79.2% of residency program directors perceived that the quality of the learning environment would somewhat or strongly decrease.28 Another study evaluating the cost of implementing the ACGME rules suggested that educational opportunities may decline because of a shift in the service–learning balance.29 Prior studies have largely relied on resident and faculty perceptions of educational experience but have not directly measured patient case mix, clinical documentation, or procedure logs. Few studies have evaluated the effects of the comprehensive changes implemented in July 2011.
We used an informatics-based approach, allowing near-complete objective capture of resident educational exposures, to evaluate the impact of the new 16-hour limitation on five key areas of intern education: (1) patient volume, (2) note characteristics, (3) exposure to common presenting problems, (4) procedural experience, and (5) structured didactic experiences. We hypothesized that, despite substantial structural changes, there would be no significant difference in any of the five areas.
We conducted the study at Vanderbilt University Medical Center (VUMC), a 900-bed quaternary care academic medical center located in Nashville, Tennessee. Approval for this study was obtained from the institutional review board.
In April 2011, Vanderbilt transitioned all inpatient internal medicine services from an overnight call system (experienced by the 2010–2011 interns) to a night float system (experienced by the 2011–2012 interns). In the overnight call system, both residents and interns worked up to 30 consecutive hours every fourth day/night; in the night float system, residents and interns were scheduled to work 13-hour shifts (16-hour maximum), with night coverage provided by an independent team (see Figure 1). We compared interns’ work and educational exposures during the first six 4-week rotations of 2010 (July 1, 2010 to December 18, 2010) and of 2011 (July 1, 2011 to December 18, 2011). During the 2010 study period, the internal medicine residency program had 121 total residents and 47 interns; during the 2011 study period, the residency had 130 total residents and 50 interns. All data for the 2010 intern group were derived from their work both on primary teams and in cross-coverage; likewise, all data for the 2011 intern group were derived from their work on both day and night teams, as shown in Figure 1. The 2011 interns each completed two weeks, or 12 shifts, of night float during the study period. For patient volume, note characteristics, common presenting problems, and procedural experience, we evaluated only inpatient experiences at VUMC. Although residents included in this study also rotate through the affiliated Veterans Administration (VA) Medical Center, experiences at the VA hospital were excluded because of incomplete capture of interns’ VA experiences. Inpatient rotations at VUMC accounted for approximately two-thirds of the time spent in intern rotations for both cohorts.
Evaluating patient volume and note characteristics
VUMC tracks resident clinical exposures via a Web-based portfolio system called Learning Portfolio (LP), which is linked to the electronic medical record (EMR) system. As students and residents write clinical notes, these notes are automatically captured by LP and indexed both by patient and trainee. LP uses these notes to create a log of patient experiences.
We evaluated the broad clinical experiences recorded in intern-generated inpatient clinical notes (specifically, admission history and physical and daily progress notes). Typically, these notes are written by only one author. We evaluated all notes written by interns in the two study periods. In 2009, VUMC developed a set of standardized electronic forms to be used for all internal medicine resident notes. In these forms, the history of present illness, subjective, physical exam, and assessment and plan sections have placeholders but do not contain any autogenerated text (i.e., all content must be entered by the note’s author). We restricted our analysis to these notes, which automatically capture all daily notes written by internal medicine trainees.
Natural language processing (NLP) tools were used to process each clinical note to identify biomedical concepts and their corresponding section locations (e.g., “history of present illness,” “cardiovascular exam”) in an automated fashion. These tools, SecTag30 and the KnowledgeMap concept indexer,31 use rigorous, score-based methods to clarify ambiguous document phrases using contextual clues (e.g., “CHF” is correctly defined as either “congestive heart failure” or “congenital hepatic fibrosis” based on the context). Concepts are matched to standard terminologies in the Unified Medical Language System, which currently contains more than eight million terms grouped into about two million concepts.32 Each concept is also identified by semantic type (e.g., “disease or syndrome,” “clinical drug,” “sign or symptom”). These NLP systems were shown to identify narrow31,33 and broad34 concepts, including tracking broad themes in trainee notes.35 We evaluated the counts of concepts covered in each note by section (e.g., history of present illness) as well as by common semantic types (e.g., diseases).
Exposure to common presenting problems
We evaluated intern exposure to five common presenting problems: fever, shortness of breath, chest pain, dysuria, and abdominal pain. These presenting problems have been previously described as measures of clinical competence in work done by Gotterer et al.36 These represent 5 of 25 presenting problems, selected by master clinical teachers and clerkship directors, in which Vanderbilt medical students are expected to demonstrate competence. When compared with the Healthcare Cost and Utilization Project Nationwide Inpatient Sample, these five presenting problems encompassed 16 of the top 25 reasons for hospital admission to internal medicine services.37 To identify these concepts, we created models based on logistic regression analysis that score documents on the basis of specific sets of clinical concepts in certain sections of the clinical notes (e.g., “cough” in a chief complaint), adapted from algorithms evaluated previously.35 Algorithms used in this study were validated by manual physician review; receiver operator curve analysis demonstrated areas that were >0.80, and each algorithm was tuned to achieve a positive predictive value of 85%.
We studied five common procedures of sufficient frequency in internal medicine to warrant comparison: arterial line placement, central line placement, thoracentesis, paracentesis, and lumbar puncture. Creation of procedure notes in the EMR is a mandatory part of clinical documentation, and entry of any procedure note automatically creates an entry in the procedure log for the trainee(s) involved.
We evaluated conference attendance with a nametag barcode swipe system at three didactic conferences: morning report conferences at VUMC or the Nashville VA Medical Center (each occurring three times weekly) and a weekly noon conference led by the chief residents. Residents are expected to attend all conferences and have an opportunity to register attendance until five minutes after the conference starts. Attendance rates were calculated as the number of actual attendees (as recorded by ID card swipe data) divided by the number of residents expected to attend the conference (based on staffing assignments).
Resident duty hours
We also compared individual resident duty hours logs before and after implementation of the 16-hour maximum shift. Duty hours were self-logged by interns in a Web-based program separate from the EMR. The average number of hours worked per day was calculated for each intern and was subsequently multiplied by six to calculate weekly work hours (consistent with ACGME regulations requiring one day off in seven). In this way, missing entries were not assumed to represent a day off. The available records were assumed to provide a representative, but not comprehensive, sample of duty hours.
We evaluated intern patient volume (including notes written and number of unique patients seen) using a two-sample t test with equal variances for normally distributed data. Intern procedure counts and NLP-derived exposure data were compared using the Wilcox rank-sum test and the binomial test of proportions. We evaluated each procedure as an outcome variable independently and as a sum of total procedures performed by each intern. Teaching conference attendance rates before and after the change in duty hours were analyzed using a binomial test of proportions. Estimated average hours worked per week were analyzed using a two-sample t test. All analyses were performed using two-tailed tests of association. We performed statistical analyses using STATA 10.1 statistical software (StataCorp, College Station, Texas).
Study group characteristics
Baseline intern characteristics from both study groups are summarized in Table 1. All 47 interns from 2010 and 50 interns from 2011 were included in the data analysis of patient volume, note characteristics, exposure to common presenting problems, and procedural experience.
When comparing the first six blocks (24 weeks) of 2010 and 2011, interns cared for more unique patients (a mean of 118 versus a mean of 140 patients per intern, P = .005) after implementation of the 16-hour maximum shift (Table 2). Interns also wrote significantly more history and physicals (a mean of 73 versus a mean of 88 per intern, P = .005) after 16-hour shift restrictions. There was no difference in the number of daily progress notes (a mean of 286 versus a mean of 298 per intern, P = .92) or overall notes (a mean of 360 versus a mean of 376 per intern, P = .51) written by interns during the study periods.
Table 3 shows the differences in clinical note content for history and physicals and progress notes between the two study periods. From the 2010 group, 3,434 history and physicals and 13,447 progress notes were analyzed. From the 2011 group, 4,443 history and physicals and 14,943 progress notes were analyzed. The word length of history and physicals increased (median of 618 words versus median of 711 words, P = .001) after implementation of the 16-hour maximum shift. Likewise, word length of progress notes also increased (median of 528 versus median of 582 words, P = .001). Interns also discussed more total concepts after the 16-hour maximum shift implementation in both history and physicals and progress notes; these increases were by 14% (338 versus 387, P = .001) and 10% (316 versus 349, P = .001), respectively. Interns discussed more unique diseases, signs/symptoms, and medications after the duty hours change; however, these differences may not be clinically meaningful (Table 3).
To further investigate this effect, we evaluated the time (by hour of day) at which admission history and physical notes were finalized into the EMR (Figure 2). In the overnight call system (2010), the times at which notes were signed were relatively evenly spaced throughout the day. After implementation of the 16-hour maximum shift (2011), most notes were signed near the ends of shifts, 5 to 7 AM for night shifts and 6 to 8 PM for day shifts. The number of concepts per note was higher at almost every hour in 2011 than in 2010, demonstrating that interns on both day and night shifts wrote more in-depth notes and that the effect was not limited to just day or night.
Exposure to common presenting problems
Interns wrote more notes matching each of the five selected topics after the duty hours change: fever (7% versus 10% of notes, P = .001), shortness of breath (30% versus 37% of notes, P = .001), dysuria (1.3% versus 2.6% of notes, P = .001), chest pain (2.0% versus 4.3% of notes, P = .001), and abdominal pain (1.8% versus 2.6% of notes, P = .024). An admission note could match more than one presenting problem.
The median number of overall procedures performed per intern did not change significantly after implementation of the 16-hour maximum shift. In the first six blocks of 2010, interns performed a median of six procedures (interquartile range [IQR] 3-10) compared with six procedures (IQR 4-8) in the first six blocks of 2011 (P = 0.94). The proportion of interns completing at least one central line placement increased significantly after implementation of the 16-hour maximum shift (62% in 2010 versus 82% in 2011, P = 0.03). The proportion of interns completing at least one of each of the other four procedures selected for study did not vary significantly between 2010 and 2011: arterial line placement (23% of interns in 2010 versus 30% in 2011, P = .46), lumbar puncture (62% versus 70%, P = .39), paracentesis (72% versus 78%, P = .52), and thoracentesis (53% versus 58%, P = .63).
Attendance at a weekly noon conference led by the chief residents increased significantly after implementation of the 16-hour maximum shift (60% of expected attendees in 2010 versus 68% in 2011, P = .001). There was no difference in attendance at morning report conferences (79% versus 78%, P = .49).
Resident duty hours
Duty hours logs were available for 44/47 (94%) interns in 2010 and 49/50 (98%) interns in 2011. The average number of weekly duty hours did not change after implementation of the 16-hour maximum shift (mean 69.0 hours per week in 2010 versus 67.8 hours per week in 2011, P = .24).
The resident educational environment has changed dramatically over the past two decades. The cornerstone of the July 2011 ACGME duty hours restrictions limited intern shift length to 16 hours or fewer, effectively eliminating overnight call. This reduction in shift length raised significant concerns regarding housestaff training in many specialties, including internal medicine. Our findings demonstrate that clinical exposures for the interns we studied did not decrease after implementing 16-hour shifts. In fact, to our surprise, interns cared for more unique patients, performed more initial workups (history and physicals), wrote longer and more detailed notes, discussed more diseases in their notes, and had higher attendance at a weekly educational conference. Meanwhile, duty hours and procedural volumes did not change. These data suggest that changes in duty hours to eliminate overnight call do not diminish intern patient care opportunities and may actually increase them.
These outcomes may have been fostered to some degree by the anticipatory changes our institution made within the Department of Medicine to ensure a smooth transition to 16-hour shifts for interns. The conversion from an overnight call system to a night float system was implemented in April 2011 with experienced residents and interns on the inpatient wards. This allowed a three-month period for schedule refinement and troubleshooting before July 2011 when the new ACGME duty hours limit took effect. In addition, resident handover education programs were expanded in anticipation of an increased number of patient handovers (Figure 1). Finally, the housestaff actively led and participated in the redesign of inpatient services, and their feedback was solicited throughout both the design and pilot phases.
Evaluating clinical exposures
Evaluating concept exposure is a new paradigm in the assessment of clinical experiences. As time spent on patient care in residency training has decreased, ensuring that housestaff have adequate exposure to key concepts becomes increasingly important. The informatics-based approach in this study allowed us to assess the breadth of concepts that residents considered in their patient evaluations. We demonstrate that limiting interns to 16-hour shifts resulted in an increase in the number of concepts considered. This was consistent across both day and night shifts, which raises several possible explanations. Interns may be more rested with limited shift lengths and write more thoughtful, comprehensive notes. Alternatively, interns may be writing broader, less focused notes. A third consideration is that interns write more in-depth notes to ensure patient safety in the setting of increased handovers/transitions of care.
Using concept exposure as an evaluative tool in resident education has broad implications. Capturing and evaluating the intellectual development during internship is complex, and it has been suggested that certain aspects of professional development, such as decision-making and procedure skills, may be better evaluated using proficiency-based tools.38,39 Concept exposure as tracked in trainee notes could be used in concert with standardized testing metrics like in-training and board examinations to more comprehensively evaluate critical thinking in medicine.
This study should be interpreted in the context of several possible limitations. Although there was no difference in average weekly duty hours per intern between the two study periods, we acknowledge the possibility of cohort bias as a limitation of this study. The individuals making up one intern class are different from those in the subsequent year. Correcting for this is challenging, however, as a randomized controlled study is likely not feasible, considering the complex logistics of residency scheduling, especially given current regulations forbidding extended shifts. Also, this is a single-institution study, similar to nearly all prior similar studies.40 Although this approach minimizes the wide variability of service structure at different institutions, as cross-institution EMRs become more commonplace, a multiprogram study may overcome this limitation.
In our study, the counts of common presenting problems likely underrepresent true exposure, as the algorithm was tuned for specificity instead of sensitivity. Similarly, the median number of procedures performed by each intern over a six-month period was lower than expected in both groups. Our methods did not capture procedures performed during inpatient time spent at the VA. Additionally, although procedure notes should be written for any procedure completed, we do not have independent observation to evaluate the fidelity of note capture. For both these measures, the relative sensitivity of the algorithm to identify exposures between different years should be similar. Finally, the system used to track duty hours was a self-reporting program separate from the one that the EMR used to track notes and procedural exposure, and it is possible that clinical experiences analyzed in the study occurred outside of reported duty hours.
Other regulatory changes
Although limiting intern shift length to 16 hours was the primary change that we evaluated in this study, several other changes were concomitantly instituted by the ACGME: Residents at the postgraduate year two level and above were limited to 24 hours of continuous duty with an additional 4 hours for patient safety and resident education, residents must have 8 hours between shifts (and should have 10 hours between shifts), and residents must not be scheduled for more than six consecutive nights of night float.3 The structural changes our institution implemented to adhere to the 16-hour restriction also accounted for the other recommended regulatory changes described above.
We found that implementation of the ACGME 16-hour rule change did not negatively affect multiple objective measures of intern clinical exposure for the interns studied. This is one of the first studies to our knowledge to offer a comprehensive, objective assessment of an internal medicine training experience, evaluating multiple spheres of intern education: patient volume, common presenting problems, breadth of concepts considered in patient workups, procedure volume, and conference attendance. Using an informatics-based approach, we provide a novel template to assess concept exposure, a critical measure to track as housestaff training time decreases. Understanding the long-term impact on knowledge base and patient care outcomes will be important in ongoing evaluation of duty hours changes.
Acknowledgments: The authors wish to thank Glenn Stein, MS, for his work on the development and maintenance of KnowledgeMap and KM Learning Portfolio, as well as Anderson Spickard III, MD, MS, and Heather Ridinger, MS, for their work in the assessment and validation of these tools. The authors also thank Tracy Dozier, MD, Eitan Freidman, MD, Waleed Khalaf, MD, Jay Montgomery, MD, Matthew Semler, MD, and Ciara Shaver, MD, for their work in the development and implementation of the resident night float system implemented at VUMC.
Funding/Support: This material is based on work supported by the Office of Academic Affiliations, Department of Veterans Affairs, VA National Quality Scholars Program, and was made possible by the use of the facilities at VA Tennessee Valley Healthcare System, Nashville, Tennessee.
Other disclosures: Funding for Dr. Vasilevskis was provided by the National Institutes of Health (K23AG040157) and the VA Tennessee Valley Geriatric Research, Education, and Clinical Center.
Ethical approval: Vanderbilt University’s institutional review board approved this study.
Disclaimer: The content of this report is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute on Aging, the National Institutes of Health, or the U.S. Department of Veterans Affairs.
4. Moonesinghe SR, Lowery J, Shahi N, Millen A, Beard JD. Impact of reduction in working hours for doctors in training on postgraduate medical education and patients’ outcomes: Systematic review. BMJ. 2011;342:d1580
5. Fletcher KE, Underwood W 3rd, Davis SQ, Mangrulkar RS, McMahon LF Jr, Saint S. Effects of work hour reduction on residents’ lives: A systematic review. JAMA. 2005;294:1088–1100
6. Jamal MH, Rousseau MC, Hanna WC, Doi SA, Meterissian S, Snell L. Effect of the ACGME duty hours restrictions on surgical residents and faculty: A systematic review. Acad Med. 2011;86:34–42
7. Barden CB, Specht MC, McCarter MD, Daly JM, Fahey TJ 3rd. Effects of limited work hours on surgical training. J Am Coll Surg. 2002;195:531–538
8. Blanchard MH, Amini SB, Frank TM. Impact of work hour restrictions on resident case experience in an obstetrics and gynecology residency program. Am J Obstet Gynecol. 2004;191:1746–1751
9. de Virgilio C, Yaghoubian A, Lewis RJ, Stabile BE, Putnam BA. The 80-hour resident workweek does not adversely affect patient outcomes or resident education. Curr Surg. 2006;63:435–439
10. Ferguson CM, Kellogg KC, Hutter MM, Warshaw AL. Effect of work-hour reforms on operative case volume of surgical residents. Curr Surg. 2005;62:535–538
11. Hutter MM, Kellogg KC, Ferguson CM, Abbott WM, Warshaw AL. The impact of the 80-hour resident workweek on surgical residents and attending surgeons. Ann Surg. 2006;243:864–871
12. Malangoni MA, Como JJ, Mancuso C, Yowler CJ. Life after 80 hours: The impact of resident work hours mandates on trauma and emergency experience and work effort for senior residents and faculty. J Trauma. 2005;58:758–761
13. Kairys JC, McGuire K, Crawford AG, Yeo CJ. Cumulative operative experience is decreasing during general surgery residency: A worrisome trend for surgical trainees? J Am Coll Surg. 2008;206:804–811
14. Durkin ET, McDonald R, Munoz A, Mahvi D. The impact of work hour restrictions on surgical resident education. J Surg Educ. 2008;65:54–60
15. Shonka DC Jr, Ghanem TA, Hubbard MA, Barker DA, Kesser BW. Four years of Accreditation Council of Graduate Medical Education duty hour regulations: Have they made a difference? Laryngoscope. 2009;119:635–639
16. Sneider EB, Larkin AC, Shah SA. Has the 80-hour workweek improved surgical resident education in New England? J Surg Educ. 2009;66:140–145
17. Gelfand DV, Podnos YD, Carmichael JC, Saltzman DJ, Wilson SE, Williams RA. Effect of the 80-hour workweek on resident burnout. Arch Surg. 2004;139:933–938
18. Welling RE, Boberg JT, Weinberg E, Gulley J. Work hours compliance in a community hospital. Curr Surg. 2004;61:241–243
19. Schneider JR, Coyle JJ, Ryan ER, Bell RH Jr, DaRosa DA. Implementation and evaluation of a new surgical residency model. J Am Coll Surg. 2007;205:393–404
20. Wong JG, Holmboe ES, Huot SJ. Teaching and learning in an 80-hour work week: A novel day-float rotation for medical residents. J Gen Intern Med. 2004;19(5 pt 2):519–523
21. Basu CB, Chen LM, Hollier LH Jr, Shenaq SM. The effect of the Accreditation Council for Graduate Medical Education duty hours policy on plastic surgery resident education and patient care: An outcomes study. Plast Reconstr Surg. 2004;114:1878–1886
22. Goldstein MJ, Kim E, Widmann WD, Hardy MA. A 360 degrees evaluation of a night-float system for general surgery: A response to mandated work-hours reduction. Curr Surg. 2004;61:445–451
23. Stamp T, Termuhlen P, Miller S, Nolan D, Hutzel P, Gilchrist J, et al. Before and after resident work hour limitations: An objective assessment of the well-being of surgical residents. Curr Surg. 2005;62:117–121
24. Gottlieb DJ, Parenti CM, Peterson CA, Lofgren RP. Effect of a change in house staff work schedule on resource utilization and patient care. Arch Intern Med. 1991;151:2065–2070
25. Afessa B, Kennedy CC, Klarich KW, Aksamit TR, Kolars JC, Hubmayr RD. Introduction of a 14-hour work shift model for housestaff in the medical ICU. Chest. 2005;128:3910–3915
26. Parthasarathy S, Hettiger K, Budhiraja R, Sullivan B. Sleep and well-being of ICU housestaff. Chest. 2007;131:1685–1693
27. McCoy CP, Halvorsen AJ, Loftus CG, McDonald FS, Oxentenko AS. Effect of 16-hour duty periods on patient care and resident education. Mayo Clin Proc. 2011;86:192–196
28. Willett LL, Arora V, Halvorsen AJ, Chacko KM, Chaudhry S, McDonald FS. Challenges with duty hour reforms: A survey of residency program directors. Am J Med. 2011;124:370–378
29. Nuckols TK, Escarce JJ. Cost implications of ACGME’s 2011 changes to resident duty hours and the training environment. J Gen Intern Med. 2012;27:241–249
30. Denny JC, Spickard A 3rd, Johnson KB, Peterson NB, Peterson JF, Miller RA. Evaluation of a method to identify and categorize section headers in clinical documents. J Am Med Inform Assoc. 2009;16:806–815
31. Denny JC, Smithers JD, Miller RA, Spickard A 3rd. “Understanding” medical school curriculum content using KnowledgeMap. J Am Med Inform Assoc. 2003;10:351–362
32. Humphreys BL, Lindberg DA, Schoolman HM, Barnett GO. The Unified Medical Language System: An informatics research collaboration. J Am Med Inform Assoc. 1998;5:1–11
33. Denny JC, Miller RA, Waitman LR, Arrieta MA, Peterson JF. Identifying QT prolongation from ECG impressions using a general-purpose natural language processor. Int J Med Inform. 2009;78(suppl 1):S34–S42
34. Denny JC, Smithers JD, Armstrong B, Spickard A 3rd. “Where do we teach what?” Finding broad concepts in the medical school curriculum. J Gen Intern Med. 2005;20:943–946
35. Denny JC, Bastarache L, Sastre EA, Spickard A 3rd. Tracking medical students’ clinical experiences using natural language processing. J Biomed Inform. 2009;42:781–789
36. Gotterer GS, Petrusa E, Gabbe SG, Miller BM. A program to enhance competence in clinical transaction skills. Acad Med. 2009;84:838–843
38. Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: A systematic review and meta-analysis. JAMA. 2011;306:978–988
39. Schroedl CJ, Corbridge TC, Cohen ER, et al. Use of simulation-based education to improve resident learning and patient care in the medical intensive care unit: A randomized trial. J Crit Care. 2012;27:219.e7–13
40. Reed DA, Fletcher KE, Arora VM. Systematic review: Association of shift length, protected sleep time, and night float with patient care, residents’ health, and education. Ann Intern Med. 2010;153:829–842