The Residency Review Committee of the Accreditation Council for Graduate Medical Education (ACGME) has put forth general program requirements, which cover certification, training, and the availability of clinical supervisors.1 These requirements inform institutional policies regarding resident supervision. Across institutions, policy statements share the common themes of patient safety, clinical care, and trainee education while also highlighting the importance of affording housestaff graduated levels of responsibility throughout the course of their training. Trainees themselves have identified inadequate supervision as one of the most common causes of the medical errors that occur during a patient's hospitalization.2–5
Specific subspecialties had, prior to the 2010 changes to duty hours regulations, defined their own levels of supervision; some specialties have required the physical presence of an attending during the delivery of key aspects of care since as early as 1990.6 In 2008, after revisiting the 2003 duty hours regulations, the Institute of Medicine (IOM) recommended in its report Resident Duty Hours: Enhancing Sleep, Supervision and Safety7 that graduate medical training programs should augment resident supervision, including on nights and weekends, through immediate access to an on-site, residency-approved supervisor. Further, the most recent iteration (2010) of the ACGME recommendations for duty hours include (in addition to further limitations in the consecutive duty hours of postgraduate year 1 [PGY-1] trainees) explicit recommendations for trainee supervision.8 These new recommendations include defining explicit requirements for trainee supervision in subspecialties not previously defined, and assessing a trainee's ability to supervise more junior colleagues before he or she actually takes on a supervisory role.
Whereas narrative reviews and observational studies have concluded that enhanced attending supervision results in improved education and patient outcomes,9–12 to date no systematic review of the literature on the effects of attending supervision has been published. Systematic review is critical to identify whether, how, and what type of supervision improves resident education and patient outcomes.13 Examining the evidence is especially important given the current debate over how to implement the above-mentioned 2010 ACGME recommendations regarding the supervision of trainees. To this end, we performed a systematic review of the literature to describe the effect of clinical supervision on residency patient- and education-related outcomes.
The 2009 PRISMA Statement on the content and reporting of systematic review manuscripts14 informed our research methods. We searched the English-language literature for studies regarding clinical supervision of housestaff (PGY-1 or greater) in U.S.- and Canadian-based training programs from 1966 (the beginning of electronic indexing of references) to 2010 using the electronic databases PubMed and JSTOR. Given the fundamental differences in the current approach to and regulation of supervision of trainees in programs throughout the world, we decided to focus exclusively on medical training programs in the United States and Canada. After consulting a medical librarian, we performed a comprehensive search in August 2009 and again in November 2010 using the following strategy: [(“Education, Medical, Graduate”[Mesh] OR “Internship and Residency”[Mesh]) AND (supervis* OR oversight)]. We conducted a hand search for journals not electronically indexed between 1966 and 2010 (e.g., Medical Education). We also included pertinent references from the studies that we included in the abstraction process. Finally, we reviewed proceedings from medical-education-focused meetings (e.g., the Association of American Medical Colleges annual meeting and the ACGME annual meeting) from 2007 through 2010, using electronic and hand searching methods.
We included studies (1) that described controlled study designs (e.g., randomized controlled trials [RCTs], concurrent control, or pre–post studies), (2) that examined the implementation of a supervision-related intervention among graduate medical education trainees (PGY-1 to highest level of postgraduate year of specialty in question) in any field of practice in any clinical setting (e.g., academic or community based; inpatient or ambulatory), and (3) that relayed patient- or education-related outcomes.
All of us reviewed the titles and abstracts of all the articles our search uncovered (see Figure 1). Four of us (L.P., M.P., S.M., and E.C.) performed the initial data abstraction of, respectively, about a quarter of the selected studies, and two of us (V.A., J.F.) reviewed those data to confirm results and resolve any discrepancies. We used a standardized abstraction tool (see Supplemental Digital Appendix 1, http://links.lww.com/ACADMED/A76) that requested the following:
* the role of the supervisor and supervisee,
* the study design and site (e.g., academic medical center [AMC]), community-based program),
* the type of supervision (see below),
* the setting (e.g., inpatient unit, ambulatory clinic),
* the year(s) of the study,
* the intervention,
* the number of participants,
* duration of follow-up, and
* supervision-related outcomes.
The data abstraction tool defined four types of supervision: (1) direct, with the supervisor physically in the presence of the patient and the trainee, (2) indirect, with the supervisor immediately available at the site of care or (3) indirect, with the supervisor available via phone or pager and able to be physically present within a short period of time, and (4) oversight, in which the supervising physician reviews the care after the care has been delivered.6,8,15
To assess the quality of study designs, we used the previously validated Medical Education Research Study Quality Instrument (MERSQI).16 The MERSQI evaluates six domains of study quality: design, sampling, type of data, validity, data analysis, and outcomes. Items are scored on an ordinal scale with a maximum 18 allowable points.16 At least one of us rated the quality of each included study, and we resolved all disagreements by consensus.
Our initial search uncovered 1,451 articles. Of those, we selected 252 for detailed review. We excluded a total of 228 studies: 112 were not research reports, 95 were observational studies, 20 were not relevant to clinical supervision, and 1 occurred outside of North America (see Figure 1). We report here on 24 English-language studies evaluating clinical supervision interventions on patient-care- and education-related outcomes in a controlled fashion. Appendix 1, organized by specialty and quality score, summarizes some of the key results of each of the individual studies that we reviewed.
Specialties represented and types of supervision evaluated
The 24 studies that met inclusion criteria represent the following specialties: psychiatry,17–21 emergency medicine,10,22–24 surgery and its subspecialties,25–30 obstetrics–gynecology,31 internal medicine,32–35 anesthesiology,36 pediatrics,37 and radiology.38,39 Of the included studies, six examined the impact of a direct supervision intervention,10,17,24,28,31,34 seven evaluated oversight,20,22,32,33,35,38,39 and four evaluated indirect models of supervision.26,29,30,36 Seven studies compared the types of supervision with one another; four evaluated direct versus oversight,18,19,21,37 and three compared direct and either type of indirect supervision.23,25,27 Only three studies were RCTs.18,20,33 The remaining studies used either a pre–post design,17,22,23,32,34,36 case–control design,29 retrospective cohort design,19,25,27,28,30,31,35,38 or prospective cohort design.10,21,24,26,37,39 Most studies were single site with small sample sizes.
Setting of supervision
Of the 24 studies we reviewed, 5 (20.8%) examined supervision in the ambulatory setting,18–20,33,34,37 and the remaining 19 (79.1%) focused on attending–resident interactions in the emergency department,10,22–24,38,39 inpatient wards,21,32,35,36 or operating room.25–31 Ambulatory studies focused on attending–trainee interactions in continuity clinic settings; one study examined ambulatory surgery experience.26 Resource utilization and patient-level variables were the focus of the inpatient, emergency department, and operating-room-based studies. Finally, only one study focused on the effect of supervision in a previously unsupervised environment—specifically, supervision in the overnight period.17
Outcomes evaluated and quality of studies
Twenty-one of the studies evaluated the effect of supervision on patients,10,18–28,30–36,38,39 and six studies17,20,29,34,35,37 evaluated educational outcomes (three of the studies examined the effect of supervision on both patients and education20,34,35). Using the MERSQI scoring criteria to assess study quality,16 scores ranged from 8.5 to 16, with a mean score of 11.8. The average MERSQI for the 21 studies evaluating patient-related outcomes was 11.9, and that for the 6 studies evaluating education-related outcomes was 11.0.
Patient-related outcomes: Overview
The 21 studies that evaluated the impact on patient-related outcomes examined complications related to diagnostic or therapeutic procedures,25–28,30,31,36 the treating physician's perception of the severity of the patient's illness,18,21,34 supervisor-initiated changes to missed or inappropriate diagnoses,22,24,38,39 greater success at continuing therapy,18,20 and resident practice patterns influenced by attending, such as resource utilization19,23,33,35 or compliance with protocols.10,32
Complications related to diagnostic or therapeutic procedures.
Eight studies evaluated the impact of a supervisory intervention on the performance of an invasive procedure.25–28,30,31,36 A retrospective cohort study of nearly 40,000 cases by Itani and colleagues25 revealed no significant difference in mortality between patient groups with and patient groups without an attending physically present, indicating that the degree of trainee supervision did not affect the overall clinical outcomes for patients, when adjusted for patient factors. A study that evaluated the impact of direct attending supervision of trainees on postoperative pain26 scores showed no significant difference between procedures performed by the attending surgeon alone versus those performed by a resident with direct attending-level supervision.
A medical record audit showed that greater attending presence was associated with significantly lower mortality and patient complication rates when procedures were stratified according to elective (e.g., plastic) surgical specialties and nonelective (e.g., trauma, cardiac) surgical specialties at a teaching hospital or AMC.27 However, Rogers and colleagues28 demonstrated no significant differences in mortality in patient groups at two tertiary care centers: one with a resident in-house (RIH) alone, and one with an in-house, board-certified attending surgeon. These researchers did note a greater chance of both technical and provider error among the RIH group.28
Lastly, from the surgery literature, a retrospective chart review of bilateral mammoplasty at an AMC showed no significant differences in complication rates between the side operated on by the attending alone versus the side operated on by the supervised resident.30
In related obstetrics–gynecology and anesthesia literature, a pre–post study of anesthesia resident intubations outside of the operating room revealed a significant decrease in intubation-related complications when attending presence was instituted.36 Finally, a medical record audit in obstetrics–gynecology revealed that the operative time for the teaching cases (i.e., those with residents and attendings) was longer (not significantly) than for the nonteaching cases and that a significantly higher incidence of seromas postoperatively occurred in the nonteaching cases.31
These studies of resident supervision in the operating room demonstrated either a positive or neutral effect on patient-specific outcomes of all types (e.g., complications, mortality, operating room time, pain) when trainees were directly or indirectly supervised by attending providers.
Physician perception of patient illness severity.
Three studies focused on the perception of an illness's severity, specifically on how attending providers rated patient acuity.18,21,34 Gennis and Gennis34 found that attendings who personally evaluated patients judged the patients to be more severely ill than did the resident. Similarly, Stein and colleagues21 noted a statistically significant difference in acuity ratings of patient illness; attendings who heard cases (but were not present for the patient interview) judged the illness to be of significantly higher acuity on validated scales than did the residents who actually (without supervision) interviewed the patients. These authors did not note differences between the acuity ratings of the resident and the attending if both were present for the resident's interview of the patient; however, the overall acuity ratings of attendings and residents (across cases) were generally higher if both the resident and attending were present for the interview (versus lower overall acuity ratings for the cases in which the resident conducted the interview without supervision). Conversely, Jaynes and colleagues18 found that residents who evaluated patients under direct attending supervision judged more patients to be “moderately ill” and fewer patients to be “severely ill” than did residents whose attendings simply provided oversight. In sum, the perception of patient acuity varied with the direct presence of the attending, with two studies demonstrating direct evaluation to result in more severe acuity ratings.21,34
Supervisor-initiated changes or missed diagnoses.
Four studies involved supervisor-initiated changes made to missed or inappropriate diagnoses: two from the radiology literature38,39 and two from emergency medicine.22,24 Albano and colleagues38 demonstrated nearly identical faculty and resident interpretations of abdominal CT scans in the setting of suspected appendicitis. Similarly, Velhamos and colleagues39 prospectively evaluated discrepancies between resident and attending interpretations of trauma-related cases, and only 11% of faculty interpretations were found to be different than the preliminary resident interpretation. Notably, the resident postgraduate year correlated with the likelihood of missed findings: the more experienced the resident, the lower the likelihood of a discrepancy between resident and faculty.39
The emergency medicine studies focused on change in patient management. Holliman and colleagues22 conducted a pre–post study, in which attendings reviewed cases after residents performed traditional patient presentations. Their results showed that attendings not only noted significant missed findings, including some that were life threatening, but also changed treatment plans in their postpresentation review of cases.22 There was no stratification for postgraduate year, although PGY-1 trainees were excluded. Finally, Sacchetti24 performed a prospective review of resident treatment plans, specifically looking for the degree of change attendings noted after they reviewed the plan; a third of cases required minor modifications, and several required major ones.
Studies examining oversight provided by attending physicians in the delivery of direct patient care did demonstrate that the attending physician frequently modified the treatment or diagnostic plan of the resident physician, to either minor or major degrees; however, when the oversight occurred in a more objective manner (e.g., review of a radiologic image), investigators noted fewer discrepancies between faculty and trainees.
Only two studies looked at the effect of supervision on patients' continuing therapy.18,20 The investigators of one RCT18 judged the patients of directly supervised residents in an ambulatory clinic in psychiatry to have overall improved status and greater success at continuing treatment. An additional RCT in psychiatry20 noted similar findings: Directly supervised residents reported higher ratings of patient outcomes on a nominal scale.
Resident practice patterns.
Six studies evaluated resident practice patterns—specifically, resource utilization19,23,33,35 and compliance with protocols10,32—as influenced by the attending supervisor. Two internal medicine studies evaluated resource utilization from the ambulatory33 and inpatient perspective.35 An RCT by Everett33 revealed that residents' outpatient laboratory testing utilization was not influenced by the practice patterns of their supervising attending physician. Feinglass and colleagues35 examined the relationship between resource use and perceived level of autonomy in a retrospective review; their results revealed that higher perceived autonomy predicted significantly lower resource use. Jin et al19 evaluated resource utilization with respect to constant observation use in the field of psychiatry. They found that residents who received training and had more intensive attending-level supervision were less likely than residents with no training and standard oversight to order constant observation for patients. Kroboth and colleagues23 performed an assessment among the admitting practices of emergency medicine residents who experienced varying degrees of supervision, and found the following among residents with greater supervision: a significant increase in hospital admission rates, a significant decrease in the use of the observation unit, and a significant decrease in patients leaving without an examination. The impact of attending presence on resource utilization is equivocal, with increased admission rates demonstrated but decreased utilization of intensive patient observation orders.
Two studies examine protocol adherence. Blumberg and colleagues32 evaluated resident compliance with inpatient protocols for high-risk patients and noted that, after the community teaching hospital initiated faculty review and discussion of the medical records that housestaff completed, resident compliance became more consistent with prescribed protocol. Sox et al10 demonstrated that directly supervised residents had a higher mean rating of compliance with the process-of-care guidelines in the emergency department setting than did other residents.
Overall, attending supervision demonstrated a positive effect in resident adherence to quality-of-care guidelines but did not appear to significantly affect resource utilization, either negatively or positively.
Education-related outcomes: Overview
Six studies evaluated the impact of supervision on education-related outcomes—specifically, attendings' perception of residents' diagnostic or procedural skills,29,34 residents' performance on standardized testing,37 residents' perception of autonomy,35 and residents' overall satisfaction.17,20
Diagnostic or procedural skills.
Two studies report the effect of supervision on diagnostic skills related to history taking, physical examination, and procedures. Gennis and Gennis34 conducted a pre–post study in which attendings rated the diagnostic skills (i.e., history taking and physical examination) of internal medicine housestaff first after hearing the resident's case presentation exclusively and, then, after examining the patient with the resident. The attendings' ratings of housestaff skills decreased significantly after attendings performed the actual patient examination.34 With respect to procedural skills, Ereso and colleagues29 performed a case–control study of the use of teleproctoring on surgical skills in general surgery residents. They compared the residents' scores on proctored and unproctored surgeries and found that all scores were significantly superior for the teleproctored surgeries.
Performance on standardized testing, perception of autonomy, and overall satisfaction.
Osborn and colleagues37 conducted a prospective study of residents who self-selected into one of three types of continuity clinics (university based, community based, or private practice) and experienced varying types of attending-level supervision relative to their location. They found that the differences in supervision did not change the residents' performance on the pediatrics in-training exam when analyzed by site.37 As mentioned previously, Feinglass and colleagues35 found, in their study of resource use and resident autonomy, that higher mean autonomy ratings were significant predictors of lower resource use and shorter length of patient stay. And, finally, in terms of the perception of the quality of the educational experience, one RCT20 compared cases reviewed by an attending with peer-reviewed cases. The investigators found no statistically significant difference in the perceived quality of peer versus attending case review.
The one study17 that examined the impact of attending-level supervision during a previously unsupervised rotation, the overnight shift, evaluated the self-reported effect of nighttime attending supervision on psychiatry residents in the emergency department of an AMC.17 Attendings were to be physically present in the emergency department in order to provide a direct, case-based educational experience and to provide assistance in the event of high patient load. Housestaff were not placed onto the night rotation until they had completed six months of training, and night duty was divided equally amongst residents. Trainees completed surveys both before and after these changes; the postimplementation surveys revealed a nonsignificant increase in comfort, a nonsignificant decrease in anxiety and isolation, and a nonsignificant increase in the sharing of responsibility.17
Costs of increased supervision
None of the studies we reviewed commented either on the cost of maintaining attending-level supervision or on the additional cost of maintaining an attending in-house in lieu of trainees independently. The study that demonstrated longer operative times when attending-level providers directly supervised residents31 did not comment on the impact of longer case time on the overall system, with respect to cost and throughput.
The results of this review indicate that enhanced attending supervision of trainees, especially in activities already supervised to at least the oversight degree, result in positive changes in patient- and education-related outcomes. Few studies showed that attending presence had no effect on specified outcomes, but two25,31 showed increased incidence of surgical complications in nonteaching environments. Faculty personally evaluating a patient, with or without the trainee present, often resulted in a change in the overall assessment of the diagnostic or therapeutic plan and, more often than not, resulted in patients being assessed as less severely ill. Missed diagnoses when attendings were absent, as well as major changes to care plans after attendings reviewed residents' work, did occur, albeit infrequently, and most studies showed that these errors and changes were related to the level of training of the resident. Lack of data and objective, standardized metrics to measure supervisory quality present challenges to making strong conclusions and represent areas for future research to better delineate the impact of supervision on trainees.
This review has implications for the current debate on the necessity of continuous, on-site, attending-level supervision initially recommended by the 2008 IOM report7 and subsequently explicitly endorsed by ACGME regulation.8 Importantly, this review does suggest that enhanced attending-level supervision, in a variety of ways, improves both patient- and education-related outcomes in the majority of subspecialties and a majority of clinical settings. The benefits for trainee education are present in both ambulatory and inpatient settings. Further, increased supervision during procedure-based scenarios generally results in improved patient-related and educational outcomes. As mentioned, our review revealed that enhancing attending-level presence in already-supervised activities has positive benefit; however, only one study17 examined the effect of introducing formalized supervision in previously unsupervised activities, such as the overnight or on-call period (the effects were not statistically significant).
The fact that, to date, only one study17 has examined the impact of enhanced nighttime supervision on patient- and education-related outcomes limits the ability to make valid conclusions regarding the specific recommendation of overnight presence of faculty. Based on the results of this review, there is not sufficient evidence to definitively support continuous, on-site, attending-level supervision for trainees; however, in the absence of such evidence, patient safety concerns should drive current decisions regarding overnight supervision practices. Additional rigorous studies will need to examine the impact of such a policy change before mandating a costly intervention with questionable efficacy.40,41
The lack of evidence for mandating enhanced supervision must be considered in the context of the current ACGME recommendations, which specify the need for access to a “qualified supervisor.”8 A “qualified” supervisor is not necessarily someone who has fully completed his or her postgraduate training; it may be a more senior resident colleague—someone who has only marginally more training than the supervisee. A small number of studies did demonstrate that the number of changes or corrections made to a patient assessment by a faculty physician depended on the postgraduate year of the trainee; more advanced trainees' clinical assessments required fewer modifications. These findings suggest that supervision is not a one-size-fits-all approach and that the degree of faculty oversight depends on the trainee's perceived and actual levels of skill. Objective criteria either to measure the readiness of a trainee to perform in the role of supervisor or to indicate the level of supervision required are lacking, as is education for supervisors of all levels. Again, this lack speaks to the limited work in assessing the effects of supervision and indicates directions for possible future endeavors, such as defining milestones to identify more senior trainees who may be prepared to supervise their more junior colleagues.
The absence of objective outcomes related to the quality of supervision limits the rigor with which the results of supervision can be explored—specifically, quantifying and qualifying the effects of clinical supervision on patient- and education-related outcomes. Although many investigators have performed observational studies of current strategies in an attempt to formulate a framework for analysis,12 no validated instrument is available either to quantify the quality of supervision (given or received) or to relate supervision to clinical outcomes. Many of the studies we reviewed relied on nonvalidated instruments or nonobjective measures to evaluate the impact of the intervention.
There are several limitations to our review including the small overall number of studies included. Of those studies included, many occurred at a single institution and, as mentioned, used a small sample size. In addition, few represented RCTs. The majority either used exclusively nonvalidated survey data or lacked objective assessments of the clinical supervision provided. Publication bias could also influence the results of our review.42 We did not conduct a search of abstracts or papers indexed within EMBASE, which may have decreased the number of studies included in the final review. Finally, we chose to focus specifically on graduate medical education and did not examine the impact or use of supervisors in other health professions such as nursing.
Despite these limitations, this work provides a review of the existing literature and helps to shape the current understanding of the effect of supervision on patient- and education-related outcomes. It suggests that enhanced attending supervision of care provided by resident physicians in a variety of clinical settings may improve both patient- and education-related outcomes. However, there is insufficient evidence at present to mandate continuous, on-site attending supervision as previously recommended by the IOM.7 Importantly, this lack of objective evidence should not cloud the ultimate goal, which is the delivery of safe, quality patient care. Future research should be conducted to establish the effects of enhanced on-site supervision, at any level, particularly during nights and weekends. Lastly, objective measures of the quality and extent of supervision provided to trainees are needed to improve the rigor and further the work in this area of research.
The authors wish to thank Dr. Kathlyn Fletcher from the Medical College of Wisconsin for her valuable insights regarding systematic review, Dr. Darcy Reed from Mayo Clinic College of Medicine for the use of the Medical Education Research Study Quality Instrument (MERSQI), and Ms. Christa M. Modschiedler, biomedical bibliographer at the John Crerar Library at the University of Chicago, for technical assistance with our search strategy.
Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A76.
2. Shojania KG, Fletcher KE, Saint S. Graduate medical education and patient safety: A busy—and occasionally hazardous—intersection. Ann Intern Med. 2006;145:592–598.
3. Jagsi R, Kitch BT, Weinstein DF, Campbell EG, Hutter M, Weissman JS. Residents report on adverse events and their causes. Arch Intern Med. 2005;16:2607–2613.
4. Conigliaro J, Frishman WH, Lazar EJ, Croen L. Internal medicine housestaff and attending physician perceptions of the impact of the New York State Section 405 regulations on working conditions and supervision of residents in two training programs. J Gen Intern Med. 1993;8:502–507.
5. Singh H, Thomas EJ, Petersen LA, Studdert DM. Medical errors involving trainees: A study of closed malpractice claims from 5 insurers. Arch Intern Med. 2007;167:2030–2036.
6. Harbison J. Housestaff Supervision Policy: A Virginia Consensus on Resident Supervision: Graduate Medical Education Committee, University of Virginia Medical Center. , 1990 [no longer available].
7. Ulmer C, Wolman DM, Johns MME, eds. Resident Duty Hours: Enhancing Sleep, Supervision, and Safety. Washington, DC: National Academies Press; 2008.
8. Nasca TJ, Day SH, Amis ES Jr; ACGME Duty Hour Task Force. The new recommendations on duty hours from the ACGME Task Force. N Engl J Med. 2010;363:e3.
9. Haun SE. Positive impact of pediatric critical care fellows on mortality: Is it merely a function of resident supervision? Crit Care Med. 1997;25:1622–1623.
10. Sox CM, Burstin HR, Orav EJ, et al.. The effect of supervision of residents on quality of care in five university-affiliated emergency departments. Acad Med. 1998;73:776–782.
11. Kilminster SM, Jolly BC. Effective supervision in clinical practice settings: A literature review. Med Educ. 2000;34:827–840.
12. Kennedy TJ, Lingard L, Baker GR, Kitchen L, Regehr G. Clinical oversight: Conceptualizing the relationship between supervision and safety. J Gen Intern Med. 2007;22:1080–1085.
13. Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: Synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126:376–380.
15. Farnan JM, Humphrey HJ, Arora V. Supervision: A 2-way street. Arch Intern Med. 2008;168:1117.
16. Reed DA, Beckman TJ, Wright SM, Levine RB, Kern DE, Cook DA. Predictive validity evidence for medical education research study quality instrument scores: Quality of submissions to JGIM's Medical Education Special Issue. J Gen Intern Med. 2008;23:903–907.
17. Finlayson AJ, Bartolucci G, Streiner DL. Deployment, supervision and decision-making of residents in an emergency psychiatric service. Can J Psychiatry. 1979;24:207–211.
18. Jaynes S, Charles E, Kass F, Holzman S. Clinical supervision of the initial interview: Effects on patient care. Am J Psychiatry. 1979;136:1454–1457.
19. Jin C, Novik S, Saravay S. Consultation–liaison psychiatry training and supervision results in fewer recommendations for constant observation. Gen Hosp Psychiatry. 2000;22:359–364.
20. Kass F, Charles E, Eagle P, Yudofsky B. Assessing the impact of supervision on outpatient evaluation in psychiatry. QRB Qual Rev Bull. 1984;10:3–5.
21. Stein SP, Karasu TB, Charles ES, Buckley PJ. Supervision of the initial interview: A study of two methods. Arch Gen Psychiatry. 1975;32:265–268.
22. Holliman CJ, Wuerz RC, Kimak MJ, et al.. Attending supervision of nonemergency medicine residents in a university hospital ED. Am J Emerg Med. 1995;13:259–261.
23. Kroboth FJ, Brown FH, Stewart R, Karpt R, Karpf M, Levey GS. Impact of attending level supervision of the emergency department experience. Ann Emerg Med. 1982;11:192–196.
24. Sacchetti A, Carraccio C, Harris RH. Resident management of emergency department patients: Is closer attending supervision needed? Ann Emerg Med. 1992;21:749–752.
25. Itani KM, DePalma RG, Schifftner T, et al.. Surgical resident supervision in the operating room and outcomes of care in Veterans Affairs hospitals. Am J Surg. 2005;190:725–731.
26. Nguyen CT, Hernandez AV, Gao T, Thomas AA, Jones JS. Office based vasectomy can be performed by supervised urological residents with patient pain and morbidity comparable to those of a staff surgeon procedure. J Urol. 2008;180:1451–1454.
27. Fallon WF Jr, Wears RL, Tepas JJ 3rd. Resident supervision in the operating room: Does this impact on outcome? J Trauma. 1993;35:556–561.
28. Rogers FB, Simons R, Hoyt DB, Shackford SR, Holbrook T, Fortiage D. In house board-certified surgeons improve outcome for severely injured patients: A comparison of two university centers. J Trauma. 1993;34:871–877.
29. Ereso AQ, Garcia P, Tsenf E, et al.. Live transference of surgical subspecialty skills using telerobotic proctoring to remote general surgeons. J Am Coll Surg. 2010;211:400–411.
30. Patel SP, Gauger PG, Brown DL, Englesbe MJ, Cederna PS. Resident participation does not affect surgical outcomes despite introduction to new techniques. J Am Coll Surg. 2010;211:540–545.
31. Akingba DH, Deniseiko-Sanses TV, Melick CF, Ellerkmann RM, Matsuo K. Outcomes of hysterectomies performed by supervised residents vs those performed by attendings alone. Am J Obstet Gynecol. 2008;199:673.e1–673.e6.
32. Blumberg ML, Joseph AM, Freeman JR. A strategy for improving the supervision and performance of moonlighting residents. Acad Med. 1995;70:155–157.
33. Everett GD. Impact of supervision by medical teachers and in-patient test control programmes on the out-patient test utilization of residents. Med Educ. 1985;19:138–142.
34. Gennis VM, Gennis MA. Supervision in the outpatient clinic: Effects on teaching and patient care. J Gen Intern Med. 1993;8:378–380.
35. Feinglass J, Schroeder J, Martin G, Wallace W, Lyons J. The relationship of residents' autonomy and use of a teaching hospital's resources. Acad Med. 1991;66:549–552.
36. Schmidt UH, Kumwilaisak K, Bittner E, George E, Hess D. Effects of supervision by attending anesthesiologists on complications of emergency tracheal intubation. Anesthesiology. 2008;109:973–977.
37. Osborn LM, Sargent JR, Willaims SD. Effects of time-in-clinic, clinical setting and faculty supervision on the continuity clinic experience. Pediatrics. 1993;91:1089–1093.
38. Albano MC, Ross GW, Ditchek JJ, et al.. Resident interpretation of emergency CT scans in the evaluation of acute appendicitis. Acad Radiol. 2001;9:915–918.
39. Velhamos GC, Fili C, Vassiliu P, Nicolaou N, Radin R, Wilcox A. Around-the-clock attending radiology coverage is essential to avoid mistakes in the care of trauma patients. Am Surg. 2001;67:1175–1177.
40. Nuckols TK, Bhattacharya J, Wolman DM, Ulmer C, Escarce JJ. Cost implications of reduced work hours and workloads for resident physicians. N Engl J Med. 2009;360:2202–2215.
41. Fletcher KE, Underwood W 3rd, Davis SQ, Mangrulkar RS, McMahon LF Jr, Saint S. Effects of work hour reduction on residents' lives: A systematic review. JAMA. 2005;294:1088–1100.
42. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA. 1990;263:1385–1389.
This study was funded by the University of Chicago Department of Medicine.
Portions of this review were presented at the 2010 University of Chicago Medical Education Day, 2010 Society of General Internal Medicine Meeting; Minneapolis, Minnesota; April 29, 2010.