Standards governing duty hours limits have generally been considered necessary in graduate medical education (GME) to protect the safety of both patients1 and residents.2 Resident sleep deprivation as a result of long duty hours has been linked to higher rates of medical errors,3 poorer clinical performance,4 adverse events,5 and attentional failures6 in observational, pre–post, and experimental studies. Longer duty hours have also been linked to resident motor-vehicle-related injuries,7 obstetric complications,8 depression,9 burnout,10 poorer quality of life11 and neuropsychological performance,12 including memory loss and reduced response times.13
In response to these safety concerns, the Accreditation Council for Graduate Medical Education (ACGME) implemented mandatory standards on July 1, 2003, that limited duty hours for medical residents in accredited U.S. GME programs.14 Although benefits from ACGME duty hours limits continue to be debated,15 few studies have described how duty hours limits may be affecting clinical training environments, trainee learning, resident access to preceptors and faculty, and resident education.16 For instance, residents have complained that mandatory duty hours rules interfere with continuity of care,17 increase cross-coverage errors,18 shift the education focus away from professionalism,19 create fear that new regulations will add additional training years,20 and cause frustration when residents are faced with heavy workloads and must reconcile actual hours against ACGME duty hours rules.21 Underscoring these concerns are the contrasting missions of the teaching hospitals, who need staff to provide professional care; faculty, who balance attending, practice, service, and research responsibilities; and residents, who need access to faculty and supervised clinical experiences to properly prepare them to enter independent practice.
To assess the effects of ACGME duty hours rules on training environments, researchers have surveyed residents using post and pre–post survey designs. Post surveys were administered after ACGME duty hours rules became mandatory. These surveys asked residents and fellows10,22–25 about their views of the success or failure of the mandatory standards. Although informative about how residents perceived duty hours limits, postsurvey results are often colored by memory loss, cohort confounds when all of the responders who have prelimits experiences have become upper-level residents by the time the survey is administered, and reporting biases when residents mimic faculty attitudes and beliefs in their survey responses.
Pre–post designs26–28 compare responses to surveys that were administered in 2003 and earlier with responses to the same surveys readministered in 2004 and later. Pre–post designs are subject to covariate biases whenever responders who took the survey prelimits (2003 and earlier) differ significantly from responders who took the survey postlimits (2004 and later). Pre–post designs are also subject to trend biases whenever naturally occurring time trends in the data confound pre–post differences. Covariate biases have been addressed by computing outcomes that are adjusted to reflect the influences of variations in responder characteristics. Trend biases are addressed by difference-in-differences methods29 where effect sizes are computed by subtracting the pre–post difference in mean responses among physician residents rotating through “effect” settings minus the pre–post difference in mean responses computed for comparable residents who rotated through “control” settings. Control settings have been identified as (1) nonteaching hospitals where duty hours limits are irrelevant,30–32 (2) training programs in teaching hospitals where duty hours limits were openly not enforced,26 or (3) responders whose duty schedules were not changed, for whatever reason, by duty hours limits. Facility-level controls are limited to outcomes that can be observed in both teaching and nonteaching settings, such as patient outcomes and medical errors, and are thus not practical for resident satisfaction surveys. Program-level controls are often difficult to implement because few program directors openly defy ACGME standards. Responder-level controls can be identified by asking respondents if the 2003 duty hours limits had any impact on their actual duty schedules. However, such questions were not answerable before 2003, when ACGME duty hours limits were first implemented. We call this the “missing-data problem.”
For this report, we introduce and apply a methodology that uses responder-level controls to assess the influence of the 2003 mandatory ACGME duty hours limits on how physician residents perceived their clinical training environments in the Department of Veterans Affairs (VA) medical centers between July 1, 2000 and June 30, 2007. The study addresses covariate confounds, trend biases, and missing-data problems in three important aspects. First, we used the Learners' Perceptions Survey (LPS), a structured interview administered annually by the VA Office of Academic Affiliations (VA-OAA) to residents rotating through VA medical centers. Second, respondents were classified into effects or control groups based on LPS survey questions that asked respondents whether duty hours limits actually changed their hours worked during scheduled VA rotations. Third, we adjusted for covariate and trend biases using a robust differencing variable technique, an advanced statistical method designed to handle the missing-data problem caused by failing to identify controls among pre-2003 responders.
We obtained resident satisfaction data from the VA LPS, which has been described elsewhere.33,34 Elements of each satisfaction domain and ACGME duty hours limits questions are listed in List 1. The analyses for this study were conducted for administrative purposes by, and were under the direct supervision of, the VA-OAA, under review by OMB Information Collection (#2900-0691) approved for VA Form #10-0439, for all data collected through January 2010.
Used as a performance metric since 2001, VA-OAA has administered the LPS annually to all trainees who rotate through VA medical centers. To reflect overall satisfaction, respondents rate “clinical training … on a scale from 0 to 100, where 100 is a perfect score and 70 is a passing score.” We dichotomized overall satisfaction responses from all available surveys (2001–2007) into satisfied (≥70) or otherwise (<70), where 70 was defined by VA as a “passing” score.
Respondents also rated each of five domains on a five-point scale (List 1). For these analyses, to compute the odds that a resident reported being satisfied, we dichotomized responses into “satisfied” (very satisfied, somewhat satisfied) and “otherwise” (neither satisfied nor dissatisfied, somewhat dissatisfied, very dissatisfied). Since 2001, domains included satisfaction with clinical faculty/preceptors, learning, working, and physical environments. A fifth domain, clinical environment, was added with the 2003 survey.
The VA added an ACGME duty hours limits question beginning with the 2004 survey (List 1). The question read, “In July 2003, the Accreditation Council for Graduate Medical Education instituted changes in requirements in duty hours/scheduling for resident education. In your opinion, what effect have these changes had on your educational experience at the VA facility…?” Respondents rated their answers on a five-point scale. To adjust pre–post differences for time trends, we constructed a differencing variable by dichotomizing responses to this question to classify each responder as either a no-effect control (response of “no effect”) or an effect-responder (response of “very positive,” “somewhat positive,” “somewhat negative,” or “very negative” effect).
Several scenarios could explain why residents may have claimed that ACGME duty hours limits had no effect on their VA clinical training settings. For example, a resident may have worked a schedule of hours that was within the duty hours limits whether or not the training program was enforcing the ACGME-mandated duty hours rules. Alternatively, the training program may have ignored duty hours rules, at least during the respondent's rotation.
To adjust for trend biases, we constructed the differencing variable to combine responders who reported either positive or negative perceptions of duty hours effects. We combined the two groups because our purpose was to measure how duty hours limits influenced residents' ratings of their training environment, and not to determine how residents actually perceived duty standards. Although important, the latter research lies outside the scope of the current study.
The LPS also obtained demographic information about each respondent, including gender, training level (postgraduate year), and specialty. For our analyses, we grouped resident specialties into medicine (internal medicine, neurology, physical and rehabilitation medicine), surgery (surgery, anesthesiology), psychiatry (including psychiatric subspecialties), and ancillary care (diagnostic specialties, radiology, pathology). Beginning in 2003, the LPS began collecting information on medical school graduation (date graduated, U.S. versus foreign medical school) and the mix of patients seen during the respondent's VA rotation. We estimated patient mix from survey responses as the percentage of patients the respondent reported seeing during “an average week, at the VA…” for each of seven patient categories: 65 years of age or older, chronic mental illness, chronic medical illness, multiple medical illnesses, alcohol/substance dependence conditions, low-income socioeconomic status, and no social/family support.
To be comparable with other studies, we computed the effect of ACGME duty hours limits on resident satisfaction for each satisfaction domain as a ratio of odds ratios (ROR) describing whether the resident reported satisfaction or otherwise. The ROR numerator is calculated for effect-responders and equals the odds that these respondents would have reported satisfaction in the postperiod divided by the odds that these same respondents would have reported satisfaction in the preperiod. The denominator is calculated in the same way, but only for no-effect controls. ROR = 1 indicates that pre–post changes in satisfaction rates among effect-responders were no different from the pre–post changes among no-effect controls, and suggests that no duty-limit effect on satisfaction was observed. On the other hand, ROR > 1 indicates that pre–post changes in satisfaction rates among effect-responders were greater than their no-effect counterparts, and suggests that duty hours limits were associated with higher satisfaction rates. Similarly, ROR < 1 suggests that duty hours limits led to decreased satisfaction rates.
ROR is adjusted for both covariate and trend biases using a robust differencing variable technique that extends difference-in-differences analyses29 to logistic regression35 by (1) using resident-level training environments as the unit of analyses, (2) identifying control respondents to adjust for trend biases, (3) accounting for missing data without imputation noise, (4) performing an exhaustive model search to adjust for covariate biases, and (5) computing estimates of effect sizes and confidence intervals (CIs) that are robust to model misspecification (see Mathematical Appendix, Supplemental Digital Content 1, http://links.lww.com/ACADMED/A19).
To account for covariate biases between pre- and postperiods, and effect- and control responders, we adjusted satisfaction outcomes based on responder characteristics and clinic experience. To account for nonlinear associations, we used a maximum-likelihood recoding strategy to transform all continuous and ordinal variables into binary covariates. Specifically, each continuous and ordinal variable was independently dichotomized using nonparametric, bootstrapped, maximum-likelihood cut-point estimates for each of the five domains and overall satisfaction score.36 For each dependent variable, we determined a model containing the most predictive covariates from an exhaustive model search37 using the generalized Akaike information criteria38 based on data from postlimits periods. We then validated these empirically motivated models using a 10-fold cross-validation approach.39
Next, we constructed a theoretically motivated model to contain three specific variables. First, a period indicator variable assumed a value of zero if the respondent answered the LPS survey in prelimits years (2001–2003), or a value of one if the respondent had answered the survey during postlimits years (2004–2007). Second, a differencing variable was constructed to assume a value of zero if the respondent was a no-effect control, and a value of one if the respondent reported either a positive or negative effect to the ACGME duty hours limits question (effect-responder). Third, a period × differencing variable interaction term was computed by multiplying the period indicator and differencing variables for each respondent.
We constructed a final model by combining the terms that made up the theoretically and empirically motivated models. All models included a constant term. We then used a nonnested model selection test40–43 to compare the fit of all three models. If the final model fit the data better than either theoretically or empirically motivated models, then duty hours limits effects were estimated by exponentiating the estimated coefficient to the period × differencing variable interaction term to the final model. To semantically interpret the interaction term, we assumed that the adjusted impact of the differencing variable on satisfaction is invariant with time (see Mathematical Appendix, Supplemental Digital Content 1, http://links.lww.com/ACADMED/A19).
The LPS did not ask respondents about duty hours limits in prelimits periods when ACGME rules were not enforced (missing-data problem). However, the concept of a no-effect control in prelimits periods is still relevant. Although one can only speculate about the actions of VA staff during 2001–2003, it is possible that some residents were assigned to work schedules that complied naturally with the duty hours rules. Residents may also have been supervised by attending physicians who would have done business as usual and ignored duty hours limits had such rules been mandatory. Assigning values to the differencing variable for prelimits responders is treated as a missing-at-random problem. That is, by knowing the year of the survey, one knows whether the value of the differencing variable is missing.44 Concerning missing data, the period × differencing variable interaction term will always equal “zero” during prelimits periods. Thus, only the differencing variable as a main effects term will have missing data. Rather than using imputation, we computed maximum-likelihood estimates by taking into account all possible patterns of values for the missing data. Missing values among covariates caused by later additions to the LPS survey were also treated in this way. Thus, model coefficients were computed directly from the observable component of the data without imputation noise.
Final models were tested for fit, the presence of model misspecification, and multicollinearity. Because of the potential for misspecification, robust estimation methods that are valid in the presence of model misspecification were used to compute both parameters and CIs.29,45,46
Table 1 presents characteristics and satisfaction scores for 19,605 LPS physician resident responders classified by reporting period. Variation in responder characteristics underscores the need to adjust for covariate biases. Compared with all residents in ACGME-accredited programs in 2008–2009,47 the LPS sample had slightly fewer females at 7,102 of 18,323 (39%) versus 48,823 of 108,176 (45%) residents in ACGME-accredited programs, had fewer international medical school graduates at 3,602 of 14,177 (25%) versus 29,488 of 108,176 (27%) ACGME residents, and fewer first-year residents at 5,498 of 19,605 (28%) versus 38,404 of 108,176 (36%) ACGME residents.
Table 2 reports estimates of duty hours limits effects measured as an ROR based on the robust differencing variable technique. The wide CIs reflect the uncertainty associated with working with incomplete datasets.
Overall, respondents tended to report higher satisfaction with their VA clinical training environment when duty hours limits applied. For instance, respondents overall were 2.46 times (95% CI [1.49, 4.05], P < .001) more likely to report satisfaction with VA as a clinical training environment under duty hours limits than without such standards. These findings held across each of the five domains, for all residents taken together, and for medicine residents only. Surgery residents tended to report higher levels of satisfaction only for clinical faculty or preceptors and clinical environment. Estimates for ancillary and psychiatry specialties were inconclusive.
To understand its relevance to education, we recalculated ROR estimates of duty hours limits effect sizes (Table 2) to reflect the adjusted estimate of the percentage of respondents who would change their response from “not satisfied” to “satisfied” as duty hours limits became mandatory (Table 3). The largest change occurred in the clinical environment domain. For surgery residents (ROR = 9.10, 95% CI [2.62, 31.61], P = .0005), satisfaction rates for clinical environments increased from a prelimits period rate of 60% (Table 1) to an expected 93% under mandatory limits, adjusted to reflect differences in the mix of respondents and other time trends in the data. That is, we estimate that 33 out of 100 respondents, who otherwise would not have been satisfied, would have reported satisfaction under mandatory duty hours limits. For medicine (ROR = 3.46, 95% CI [1.37, 8.70], P = .0084), the prelimits period satisfaction rate of 58% increased to 83%, for an adjusted net increase of 25% under the mandatory duty hours rules. Similarly, these data suggest that an expected 12% more surgery residents and 11% more medicine residents would have reported satisfaction with faculty or preceptors under ACGME mandatory duty hours rules than without such rules.
To show the importance of adjusting for covariate mix and time trends, the unadjusted pre–post period change in satisfaction with VA training environments is OR = 1.00 (95% CI [0.91, 1.11], P = .96). There was also little adjusted cross-sectional difference in overall satisfaction between effect-responders and no-effect controls (OR = 1.12, P = .66). This finding is comparable with those of studies showing few differences in patient outcomes between teaching and nonteaching VA hospitals.48 Females were generally more likely to report overall satisfaction for VA training (OR = 1.12, P = .038) as well as clinical (OR = 1.09, P = .039) and working (OR = 1.15, P < .001) environments. The higher rates of satisfaction among females are consistent with other surveys.49 Respondents who reported that 50% or more of the patients they saw were without family support, or were substance abusers, were only 56% (OR = 0.56, P < .0001) and 73% (OR = 0.73, P < .0001), respectively, as likely to report satisfaction with VA clinical training environments as their counterparts who saw fewer than 50% of such patients.
Using advanced statistical techniques to adjust for trend and covariate biases, we found that the 2003 ACGME standards significantly and materially enhanced learning satisfaction rates for medicine and surgery residents rotating through VA medical centers. The statistical tools, along with our large sample size and robust survey, provided a comprehensive estimate of the impact of duty hours limits on residents' satisfaction with their educational environment. Understanding these effects can provide useful information to government agencies, accrediting bodies, teaching hospitals, and program directors in assessing the effects of duty hours limits, and to understand how residents' satisfaction with their training environments can improve as duty hours limits rules are enforced.
These findings were consistent with subanalyses conducted across domain elements, and when satisfaction scales were “cut” at different levels. However, our results both compared to and contrasted with those of previous studies. Specifically, these findings are consistent with reported associations between reduced work hours and residents' perceptions of more time to read and learn independently,24,28,50 greater attending supervision,28,51 and attending physicians' increased role in patient care.52 In contrast, these findings differ from postsurveys22–25,53 and pre–post surveys26–28 that reported clinical experiences and patient-care quality remained unchanged, or even worsened, with fewer duty hours.
There are several possible reasons for the disparity between these survey findings and ours. First, the robust differencing variable technique applied here was designed to adjust for time trends using respondent-level controls with pre–post survey data. Such corrections, in fact, had an important effect on our study findings. For example, we found no ACGME duty hours limits effect on satisfaction rates (OR = 1.00, 95% CI [0.91, 1.11], P = .96) with LPS data when effect sizes were based entirely on unadjusted pre–post differences. Adjusting for time trends alone, the estimated effect size increased to an ROR of 2.13 (95% CI [1.27, 3.58], P = .004), and to 2.46 (95% CI [1.49, 4.05], P < .001) (Table 2) when estimates were further adjusted to account for differences in responder mix across periods and duty hours limits effect settings.
A second explanation for the discrepancies may involve differences in survey designs. For purposes of identifying control respondents, the LPS survey asked responders to rate satisfaction about current clinical rotations and whether duty hours limits (including limits on schedules and shifts) had an effect (good or bad) on the respondent's actual VA training environment. Postsurvey designs often focused on previous clinical training experiences and actual hours worked, which are subject to underreporting biases.54
A third difference may be attributable to the sample and the survey design. One-third of the nation's residents rotate through VA medical centers under VA affiliation agreements with 107 U.S. medical schools,55 with VA second only to Medicare and Medicaid as the largest funder of residency training in the United States.56 Although VA teaching medical centers likely differ from non-VA teaching hospitals, this is the largest survey of physician resident satisfaction to date and involves a variety of facility sizes and medical school affiliations in diverse geographic areas across the United States. Furthermore, the confidential LPS survey is administered by a federal agency under strict rules of confidentiality enforced under federal oversight by the Office of Management and Budget. Promoted as an administrative tool designed to improve VA as a clinical training environment,33,34 the LPS survey began with the 2001 academic year, three years before duty hours limits were first implemented, and one full year after full implementation of VA's quality improvement initiatives had been completed.57,58
Fourth, by classifying respondents individually into “effect” respondents and “no-effect” controls, we avoided aggregation errors created when respondents were grouped by educational program or facility. Overall, 36% of LPS respondents claimed that duty hours limits did not impact their VA clinical rotations during postlimits academic years (2004–2007). Such reports occurred across programs, specialties, and facilities, indicating the diversity of experiences residents encountered within the same programs and teaching facilities.
Finally, it may not be unusual to find “no-effect” environments after 2003 because some training programs had failed on occasion to adhere to mandatory duty hours rules. In one study, respondents reported exceeding the 80-hour rule at least once during six months in surgical (89%) and nonsurgical (74%) specialties while underreporting their work hours to their program directors (73% and 38%, respectively).54 In a national survey of interns after ACGME implementation, 67% reported working shifts beyond the 30-hour rule, 43% more than the 80-hour rule, and 44% less than the one-in-seven day rule.59 Despite having regulated resident duty hours since 1989, New York State found 54 of the state's 82 teaching hospitals were in noncompliance.60
The present study has certain limitations. VA clinic rotations may not necessarily represent experiences at non-VA locations. Second, respondents may not know when duty hours limits affected their training environments, thus leading to overreporting of “no effect” on the ACGME duty hours limits question. However, overreporting “no-effect” would bias estimates of duty hours limits effect sizes toward zero. Third, it is unknown whether resident satisfaction with clinical training is related to objective measures of education outcomes, such as in-service competencies examinations, board scores, and attending physician evaluations. Fourth, covariates we used to adjust for differences in respondent mix may not have controlled for all relevant factors that drive satisfaction rates. The study did not address why satisfaction may have changed, but this shift could be explained by many factors in addition to duty hours limits, including changes in workload, work life,61 resident cross-coverage, night-float systems, redistribution of workload, reassignment of noneducational tasks to midlevel and lower-level providers,62 clinical schedules that minimize sleep interruption,63 or reduced in-house on-call duties. Fifth, the results are based on resident perceptions and may not necessarily reflect true differences in the quality of patient care or the effectiveness of the teaching environment. Finally, it is unknown whether further restrictions on duty schedules will continue to improve resident satisfaction.
In summary, applying advanced statistical methods to robust survey data, we found the 2003 ACGME mandatory duty hours limits were associated with improved training satisfaction rates. With the prospect that ACGME may adopt new standards for resident duty hours,16 education researchers may wish to consider using the LPS survey design with robust differencing analyses to assess the impact of new standards across U.S. teaching hospitals.64
Sincere gratitude is expressed for the very generous review, academic direction, and support from Malcolm Cox, MD, chief academic affiliations officers, and Karen M. Sanders, MD, deputy chief academic affiliations officer, Veterans Health Administration, Department of Veterans Affairs (Washington, DC).
Gratitude is also expressed for the guidance and direction from Tetyana K. Kashner, MD, physician resident at the Pennsylvania State University Hershey School of Medicine (Hershey, Pennsylvania); Linda Godleski, MD, associate professor of psychiatry at the Yale University School of Medicine (West Haven, Connecticut); Catherine P. Kaminetzky, MD, assistant professor of medicine at the Duke University School of Medicine (Durham, North Carolina); Susan Kirsh, MD, associate professor of medicine at Case Western Reserve University School of Medicine (Cleveland, Ohio); and Edward H. Livingston, MD, professor of surgery at the University of Texas Southwestern Medical Center (Dallas, Texas).
Gratitude is also expressed for administrative and data management support from Keith Hoffman, database administrator at the Veterans Health Administration Allocation Resource Center (Braintree, Massachusetts); Robert S. Hinson, executive assistant, and Linda McInturff, Evert Melander, Cynthia Miller, and Dilpreet Singh from Veterans Health Administration Office of Academic Affiliations (Washington, DC); and Christopher T. Clarke, director, and David S. Bernett, Terry V. Kruzan, George E. McKay, and Laura Stefanowycz from the Department of Veterans Affairs Office of Academic Affiliations Data Management Center (St. Louis, Missouri).
This study was funded in part by the Small Business Innovation Research (SBIR) program from the National Cancer Institute (NCI) (R44CA139607; PI: S.S. Henley) and the National Institute on Alcohol Abuse and Alcoholism (NIAAA) (R43AA014302, R43AA013670, R43/44AA013768, R43/44AA013351, R43/44AA011607; PI: S.S. Henley), and by the Department of Veterans Affairs' Health Services Research and Development Service (SHP #08-164; PI: T.M. Kashner).
The analyses for this study were conducted for administrative purposes by, and were under the direct supervision of, the Office of Academic Affiliations, Veterans Health Administration, Department of Veterans Affairs (VA-OAA), under review by OMB Information Collection (#2900-0691) approved for VA Form #10-0439, for all data collected through January 2010.
The opinions expressed herein do not necessarily reflect the views of the Department of Veterans Affairs or its affiliates, the National Cancer Institute, or the National Institute of Alcohol Abuse and Alcoholism.
1 Fletcher KE, Davis SQ, Underwood W, Mangrulkar RS, McMahon Jr LF, Saint S. Systematic review: Effects of resident work hours on patient safety. Ann Intern Med. 2004;141:851–857.
2 Woodrow SI, Segouin C, Armbruster J, Hamstra SJ, Hodges B. Duty hours reforms in the United States, France and Canada: Is it time to refocus our attention on education? Acad Med. 2006;81:1045–1051.
3 Landrigan CP, Rothschild JM, Cronin JW, et al. Effect of reducing interns' work hours on serious medical errors in intensive care units. N Engl J Med. 2004;351:1838–1848.
4 Veasey S, Rosen R, Barzansky B, Rosen I, Owens J. Sleep loss and fatigue in residency training: A reappraisal. JAMA. 2002;288:1116–1123.
5 Szklo-Coxe M. Are residents' extended shifts associated with adverse events? PloS Med. 2006;3:2194–2196.
6 Barger LK, Ayas NT, Cade BE, et al. Impact of extended-duration shifts on medical errors, adverse events, and attentional failures. PloS Med. 2006;3:2440–2448.
7 Barger LK, Cade BE, Ayas NT, et al. Extended work shifts and the risk of motor vehicle crashes among interns. N Engl J Med. 2005;352:125–134.
8 Grunebaum A, Minkoff H, Blake D. Pregnancy among obstetricians: A comparison of births before, during and after residency. Am J Obstet Gynecol. 1987;157:79–83.
9 Reuben DB. Psychologic effects of residency. South Med J. 1983;76:380–383.
10 Golub JS, Weiss PS, Ramesh AK, Ossoff RH, Johns MM 3rd. Burnout in residents of otolaryngology–health and neck surgery: A national inquiry into the health of residency training. Acad Med. 2007;82:596–601.
11 Fletcher KE, Underwood W 3rd, Davis SQ, Mangrulkar RS, McMahon LF Jr, Saint S. Effects of work hour reduction on residents' lives: A systematic review. JAMA. 2005;294:1088–1100.
12 Rollinson DC, Rathlev NK, Moss M, et al. The effects of consecutive night shifts on neuropsychological performance of interns in the emergency department: A pilot study. Ann Emerg Med. 2003;41:400–406.
13 Hart RP, Buchsbaum DG, Wade JB, Hamer RM, Kwentus JA. Effect of sleep deprivation on first-year residents' response times, memory and mood. J Med Educ. 1987;62:940–942.
14 Philibert I, Friedmann P, Williams WT; ACGME Work Group on Resident Duty Hours. Accreditation Council for Graduate Medical Education. New requirements for resident duty hours. JAMA. 2002;288:1112–1114.
15 Yoon HH. Adapting to duty-hour limits—Four years on. N Engl J Med. 2007;356:2668–2670.
16 Ulmer C, Wolman DM, Johns MME, eds. Resident Duty Hours: Enhancing Sleep, Supervision, and Safety. Washington, DC: Institute of Medicine; 2008.
17 Okie S. An elusive balance—Residents' work hours and the continuity of care. N Engl J Med. 2007;356:2665–2667.
18 Petersen LA, Brennan TA, O'Neil AC, Cook EF, Lee TH. Does housestaff discontinuity of care increase the risk of preventable adverse events? Ann Intern Med. 1994;121:866–872.
19 Ratanawongsa N, Bolen S, Howell EE, Kern DE, Sisson SD, Larriviere D. Residents' perceptions of professionalism in training and practice: Barriers, promoters and duty hour requirements. J Gen Intern Med. 2006;21:758–763.
20 Gopal RK, Carreira F, Baker WA, et al. Internal medicine residents reject longer and gentler training. J Gen Intern Med. 2007;22:102–106.
21 Choi D, Dickey J, Wessel K, Girard DE. The impact of the implementation of work hour requirements on residents' career satisfaction, attitudes and emotions. BMC Med Educ. 2006;6:53–58.
22 Biller CK, Antonacci AC, Pelletier S, et al. The 80-hour work guidelines and resident survey perceptions of quality. J Surg Res. 2006;135:275–281.
23 Kort KC, Pavone LA, Jensen E, Haque E, Newman N, Kittur D. Resident perceptions of the impact of work-hour restrictions on health care delivery and surgical education: Time for transformational change. Surgery. 2004;136:861–871.
24 Lin GA, Beck DC, Steward A, Garbutt JM. Resident perceptions of the impact of work hour limitations. J Gen Intern Med. 2007;22:969–975.
25 Myers JS, Bellini LM, Morris JB, et al. Internal medicine and general surgery residents' attitudes about the ACGME duty hours regulations: A multicenter study. Acad Med. 2006;81:1052–1058.
26 Jagsi R, Shapiro J, Weissman JS, Dorer DJ, Weinstein DF. The educational impact of ACGME limits on resident and fellow duty hours: A pre–post survey study. Acad Med. 2006;81:1059–1068.
27 Goitein L, Shanafelt TD, Wipf JE, Slatore CG, Back AL. The effects of work-hour limitations on resident well-being, patient care, and education in an internal medicine residency program. Arch Intern Med. 2005;165:2601–2606.
28 Lund KJ, Teal SB, Alvero R. Resident job satisfaction: One year of duty hours. Am J Obstet Gynecol. 2005;193:1823–1826.
29 Bertrand M, Duflo E, Mullainathan S. How much should we trust differences-in-differences estimates? Q J Econ. 2004;119:249–275.
30 Horwitz LI, Kosiborod M, Lin Z, Krumholz HM. Changes in outcomes for internal medicine inpatients after work-hour regulations. Ann Intern Med. 2007;147:97–103.
31 Shetty KD, Bhattacharya J. Changes in hospital mortality associated with residency work-hour regulations. Ann Intern Med. 2007;147:73–80.
32 Volpp KG, Rosen AK, Rosenbaum PR, et al. Mortality among patients in VA hospitals in the first 2 years following ACGME resident duty hour reform. JAMA. 2007;298:984–992.
33 Keitz SA, Holland GJ, Melander EH, Bosworth HB, Pincus SH. The Veterans Affairs Learners' Perception Survey: The foundation for educational quality improvement. Acad Med. 2003;78:910–917.
34 Cannon GW, Keitz S, Holland G, et al. Factors determining medical student and resident satisfaction during VA-based training: Findings from the VA Learners' Perception Survey. Acad Med. 2008;83:611–620.
35 Mullahy J. Interaction Effects and Difference-in-Difference Estimation in Loglinear Models. Cambridge, Mass: National Bureau of Economic Research; 1999. Technical working paper #245.
36 Efron B, Tibshirani RJ. An Introduction to the Bootstrap. New York, NY: Chapman & Hall; 1998.
37 Furnival GM, Wilson RW. Regression by leaps and bounds. Technometrics. 1974;16:499–511.
38 Bozdogan H. Akaike's information criterion and recent developments in information complexity. J Math Psychol. 2000;44:62–91.
39 Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning. New York, NY: Springer-Verlag; 2001.
40 Rivers D, Vuong Q. Model selection tests for nonlinear dynamic models. Econom J. 2002;5:1–39.
41 Vuong QH. Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica. 1989;57:307–333.
42 Golden RM. Statistical tests for comparing possibly misspecified and nonnested models. J Math Psychol. 2000;44:153–170.
43 Golden RM. Discrepancy risk model selection test theory for comparing possibly misspecified or nonnested models. Psychometrika. 2003;68:229–249.
44 Molenberghs G, Beunckens C, Sotto C, Kenward MG. Every missingness not at random model has a missingness at random counterpart with equal fit. J R Stat Soc Series B Stat Methodol. 2008;70:371–388.
45 White H. Maximum likelihood estimation of misspecified models. Econometrica. 1982;50:1–25.
46 Golden RM, Henley SS, White H, Kashner TM. Maximum likelihood estimation for misspecified models with missing data: Theory. Manuscript, 2010.
47 Brotherton SE, Etzel SI. Graduate medical education, 2008–2009. JAMA. 2009;302:1357–1372.
48 Khuri SF, Najjar SF, Daley J, et al. Comparison of surgical outcomes between teaching and nonteaching hospitals in the Department of Veterans Affairs. Ann Surg. 2001;234:370–383.
49 Zonia SC, LaBaere RJ, Stommel M, Tomaszewski DD. Resident attitudes regarding the impact of the 80-duty hours work standards. J Am Osteopath Assoc. 2005;105:307–313.
50 Vaughn DM, Stout CL, McCampbell BL, et al. Three-year results of mandated work hour restrictions: Attending and resident perspectives and effects in a community hospital. Am Surg. 2008;74:542–546.
51 Kaafarani HM, Itani KM, Petersen LA, Thornby J, Berger DH. Does resident hours reduction have an impact on surgical outcomes? J Surg Res. 2005;126:167–171.
52 Harrison R, Allen E. Teaching internal medicine residents in the new era: Inpatient attending with duty hour regulations. J Gen Intern Med. 2006;21:447–452.
53 Coverdill JE, Adrales GL, Finlay W, et al. How surgical faculty and residents assess the first year of the Accreditation Council for Graduate Medical Education duty-hour restrictions: Results of a multi-institutional study. Am J Surg. 2006;191:11–16.
54 Carpenter RO, Austin MT, Tarpley JL, Griffin MR, Lomis KD. Work-hour restrictions as an ethical dilemma for residents. Am J Surg. 2006;191:527–532.
55 Leeman J, Kilpatrick K. Inter-organizational relationships of seven Veterans Affairs Medical Centers and their affiliated medical schools: Results of a multiple-case-study investigation. Acad Med. 2000;75:1015–1020.
56 Chang BK, Murawsky J, Feldman J, et al. Resident education in VA outpatient clinics: Continuity and advanced clinic access implementation. Fed Practitioner. 2007;24:35–36, 39–41, 44–46, 54.
57 Asch SM, McGlynn EA, Hogan MM, et al. Comparison of quality of care for patients in the Veterans Health Administration and patients in a national sample. Ann Intern Med. 2004;141:938–945.
58 Jha AK, Perlin JB, Kizer KW, Dudley RA. Effect of the transformation of the Veterans Affairs Health Care System on the quality of care. N Engl J Med. 2003;348:2218–2227.
59 Landrigan CP, Barger LK, Cade BE, Ayas NT, Czeisler CA. Interns' compliance with Accreditation Council for Graduate Medical Education work-hour limits. JAMA. 2006;296:1063–1070.
60 State Health Department Cites 54 Teaching Hospitals for Resident Working Hour Violations. Albany, NY: New York State Department of Health; June 26, 2002.
61 Vidyarthi AR, Katz PP, Wall SD, Wachter RM, Auerbach AD. Impact of reduced duty hours on residents' educational satisfaction at the University of California, San Francisco. Acad Med. 2006;81:76–81.
62 Whang EE, Mello MM, Ashley SW, Zinner MJ. Implementing resident work hour limitations: Lessons from the New York State experience. Ann Surg. 2003;237:449–455.
63 Ogden PE, Sibbitt S, Howell M, et al. Complying with ACGME resident duty hours restrictions: Restructuring the 80-hour workweek to enhance education and patient safety at Texas A&M/Scott & White Memorial Hospital. Acad Med. 2006;81:1026–1031.
64 Byrne JM, Loo LK, Giang D. Monitoring and improving resident work environment across affiliated hospitals: A call for a national resident survey. Acad Med. 2009;84:199–205.