Skip Navigation LinksHome > April 2013 - Volume 116 - Issue 4 > The Association Between Frequency of Self-Reported Medical E...
Anesthesia & Analgesia:
doi: 10.1213/ANE.0b013e318277dd65
Economics, Education, and Policy: Research Reports

The Association Between Frequency of Self-Reported Medical Errors and Anesthesia Trainee Supervision: A Survey of United States Anesthesiology Residents-in-Training

De Oliveira, Gildasio S. Jr. MD, MSCI; Rahmani, Rod BS; Fitzgerald, Paul C. MS; Chang, Ray BS; McCarthy, Robert J. PharmD

Free Access
Article Outline
Collapse Box

Author Information

From the Department of Anesthesiology, Northwestern University, Chicago, Illinois.

Accepted for publication, October 5, 2012.

The authors declare no conflicts of interest.

Address correspondence to Gildasio S. De Oliveira, Jr., MD, MSCI, Department of Anesthesiology, Northwestern Memorial Hospital, 251 E. Huron St., F5-704, Chicago, IL 60611. Address e-mail to G-jr@northwestern.edu.

Collapse Box

Abstract

BACKGROUND: Poor supervision of physician trainees can be detrimental not only to resident education but also to patient care and safety. Inadequate supervision has been associated with more frequent deaths of patients under the care of junior residents. We hypothesized that residents reporting more medical errors would also report lower quality of supervision scores than the ones with lower reported medical errors. The primary objective of this study was to evaluate the association between the frequency of medical errors reported by residents and their perceived quality of faculty supervision.

METHODS: A cross-sectional nationwide survey was sent to 1000 residents randomly selected from anesthesiology training departments across the United States. Residents from 122 residency programs were invited to participate, the median (interquartile range) per institution was 7 (4–11). Participants were asked to complete a survey assessing demography, perceived quality of faculty supervision, and perceived causes of inadequate perceived supervision. Responses to the statements “I perform procedures for which I am not properly trained,” “I make mistakes that have negative consequences for the patient,” and “I have made a medication error (drug or incorrect dose) in the last year” were used to assess error rates. Average supervision scores were determined using the De Oliveira Filho et al. scale and compared among the frequency of self-reported error categories using the Kruskal-Wallis test.

RESULTS: Six hundred four residents responded to the survey (60.4%). Forty-five (7.5%) of the respondents reported performing procedures for which they were not properly trained, 24 (4%) reported having made mistakes with negative consequences to patients, and 16 (3%) reported medication errors in the last year having occurred multiple times or often. Supervision scores were inversely correlated with the frequency of reported errors for all 3 questions evaluating errors. At a cutoff value of 3, supervision scores demonstrated an overall accuracy (area under the curve) (99% confidence interval) of 0.81 (0.73–0.86), 0.89 (0.77–0.95), and 0.93 (0.77–0.98) for predicting a response of multiple times or often to the question of performing procedures for which they were not properly trained, reported mistakes with negative consequences to patients, and reported medication errors in the last year, respectively.

CONCLUSIONS: Anesthesiology trainees who reported a greater incidence of medical errors with negative consequences to patients and drug errors also reported lower scores for supervision by faculty. Our findings suggest that further studies of the association between supervision and patient safety are warranted. (Anesth Analg 2013;116:892–7)

Clinical supervision has an extremely important role in academic medicine but it is the least examined feature of resident education.1,2 The supervision of trainees is not only a key instrument for resident learning experience but it also assures high standards of patient care and safety. Supervision provides medical residents the chance to incorporate the virtues of supervisors who can serve as role models.3 Effective supervisors provide their supervisees direction and constructive feedback whereas ineffective supervisors are rigid and often fail to instruct.4,5

Inadequate supervision can be detrimental not only to resident education but also to patient care and safety.6,7 Inadequate supervision has been associated with increased deaths of patients under the care of junior residents.8 Although the consequences of the lack of supervision have been demonstrated,6 it is unknown whether the quality of supervision received by anesthesiology residents is associated with the frequency of self-reporting of errors.

We hypothesized that residents-in-training that self-report more medical errors would report perceived lower quality of supervision than the respondents with better-perceived supervision. The primary objective of this current investigation was to evaluate an association between the anesthesiology residents-in-training self-reported frequency of medical errors and perceived quality of faculty supervision.

Back to Top | Article Outline

METHODS

The study was approved by the Northwestern University IRB. The mailing and e-mail list of United States anesthesiology trainees was obtained from the American Society of Anesthesiologists directory available to members. A database (PostgreSQL version 9.4.1) was constructed from the list and a random cross-sectional sample of 1000 residents was selected. The survey was created using Survey Monkey software (SurveyMonkey Inc., Portland, OR). To assure confidentiality of the participants, the survey was set up to delink the responses to the respondents e-mail address, but retain the internet protocol address of the respondents. The software uses an internal tracking system to allow only 1 response per survey invitation and generate a list of nonresponders. The participants who did not respond to the electronic questionnaire were mailed a copy of the survey with a return envelope addressed to the primary investigator. Mailed surveys did not contain subject identifiers.

The questionnaire was divided into 4 parts and included 21 questions. Multiple-choice questions were used. Likert scales were used to quantify respondents’ level of agreement with a statement. The first 5 questions were designed to capture characteristics of the respondents, including age, sex, number of residents in their class, year of training, and number of hours worked per week.

The second part of the survey included all 9 questions from the De Oliveira Filho et al. instrument specifically developed to examine anesthesiology residents’ perception of quality of faculty supervision.9 Each question represents a dimension of supervision (Table 1). The instrument uses a 4-point Likert scale (never = 1, rarely = 2, frequently = 3, and always = 4). The supervision score was calculated as the average of the individual responses to the 9 questions.9 The instrument has been demonstrated to have very good psychometric properties (Cronbach α coefficient = 0.93; G and φ coefficients = 0.93).7

Table 1
Table 1
Image Tools

The third part of the survey evaluated frequency of self-reported errors using 3 questions developed by previous investigators and used in other medical specialties but with applicable relevance to anesthesiology.10,11 Frequency was evaluated using a 5-point Likert scale (5 = often, 4 = multiple times, 3 = a couple of times, 2 = once, and 1 = never). The 3 questions pertained to the following statements: “I perform procedures for which I am not properly trained,” “I have made mistakes that have negative consequences for the patient,” and “I have made medication errors (dose or incorrect drug) in the last year.”

The fourth part of the survey represented the trainee’s level of agreement to causes that contribute to poor supervision from anesthesia attendings using a 5-point Likert scale (5 = strongly agree, 4 = agree, 3 = neither agree nor disagree, 2 = disagree, and 1 = strongly disagree). Questions pertained to “lack of interest for teaching from the attending” and “lack of time due to excessive clinical work.”

The primary outcome was the supervision score within the categories of frequency of self-reported error occurrence. In a prior study examining the use of the supervision score evaluation, there was a difference of 1.3 ± 0.5 between supervision that was believed to be positive compared with that which was deemed negative.9 We hypothesized that a similar difference would be present in respondents reporting errors to have occurred multiple times or often (Likert score 4 or 5), compared with those that reported a frequency of error rate of never (Likert score 1). We also anticipated a 50% response to the survey. Based on these estimates and assuming even spacing of the median response across the categories, a sample of 1000 with a 50% response rate divided in groups representing 40% (Likert score 1), 30% (Likert score 2), 25% (Likert score 3), and 5% (Likert score 4 or 5) of the sample would achieve a power of 1 to detect an any-pair difference but a power of 0 to detect an all-pair difference using the Kruskal-Wallis test with a targeted α of 0.01 and an actual α of 0.005. The average within-group standard deviation was assumed to be 0.5. These results are based on 2000 Monte Carlo samples from the null distribution and the alternate distribution.

Supervision scores were compared among the frequency of self-reported error categories using the Kruskal-Wallis test. Because of the low number (<2%) of respondents reporting errors occurring often to any of the 3 questions, the multiple and often responses were grouped together. Post hoc comparisons between categories were made using the Dunn test corrected for 12 comparisons. The median shift and confidence intervals (CIs) between categories were determined using the Wilcoxon exact procedure. A receiver operating characteristic analysis was performed to evaluate the relationship between supervision scores and the likelihood of a response of often/multiple times (Likert 4 or 5) for each of the 3 questions regarding errors. Sensitivity, specificity, and likelihood ratio of the supervision score for predicting a response of often/multiple times was calculated at a supervision score value of 3 (mean response of frequently).

The difference in supervision scores by respondent characteristic (age, sex, years of training, residency class size, and work hours) was determined using the Kruskal-Wallis or Wilcoxon rank sum test. The association of perceived barriers to effective supervision was assessed by examining the frequency of agreement or disagreement with the statement among the respondents. A P value <0.004 was considered significant for the primary outcome and CIs are reported at 99.6%; all other comparisons were considered significant at P < 0.01 and CIs are reported at 99%. Statistical analysis was performed using R version 2.15.0 (release date March 30, 2012; The R Foundation for Statistical Computing, www.r-project.org).

Back to Top | Article Outline

RESULTS

Residents from 122 residency programs were invited to participate, the median (interquartile range) per institution was 7 (4–11). Six hundred four of the 1000 anesthesiology residents invited to participate responded to the survey, 412 via the electronic version and 192 responded through the mail. The median combined response rate was 60.4%. Respondents answered 12,580 of the 12,684 data elements (99%). Questionnaires with missing responses were excluded from analysis.

Among respondents, 350 (58%) were men, 306 (51%) were younger than 30 years, and 514 (85%) worked ≤70 hours per week. The median (interquartile range) supervision score for the sample was 3.2 (2.9–3.6). The association between respondent characteristics and supervision scores is shown in Table 2. Respondents in clinical anesthesia year 2 (CA-2) reported lower supervision scores compared with CA-1 and CA-3 respondents. Supervision scores were inversely correlated with the number of work hours reported per week: ρ = −0.24 (99% CI −0.35 to −0.13, P < 0.001). This effect was observed for all 9 of the questions related to perceived supervision.

Table 2
Table 2
Image Tools

Forty-five (7.5%) of the respondents reported performing procedures for which they were not properly trained, 24 (4%) reported mistakes with negative consequences to patients, and 16 (3%) reported medication errors in the last year having occurred multiple times or often (Likert 4 or 5). Responders reporting errors occurring multiple times or often to each of the 3 questions regarding errors were from at least 10 different anesthesiology training programs. Supervision scores were inversely associated with the frequency of reported errors for the 3 questions evaluating resident-reported errors (Fig 1). The receiver operating characteristics for supervision scores for predicting a response of multiple times or often to any of the 3 questions evaluating self-reported errors are show in Table 3.

Table 3
Table 3
Image Tools
Figure 1
Figure 1
Image Tools

Twenty-three percent of respondents who reported working ≥70 hours per week reported errors multiple times or often to the question regarding performing procedures for which they were not properly trained compared with 5% of the respondents who reported working <70 hours per week (difference, 99% CI 18% (10%–25%), P<0.001). Twelve percent of CA-2 respondents reported performing procedures for which they were not properly trained, 7% reported mistakes with negative consequences to patients, and 5% reported medication errors in the last year having occurred multiple times or often. The difference in the reported error rate of multiple times or often by CA-2 respondents represented an increase (99% CI) of 12% (6%–17%, P < 0.001), 5% (1%–9%, P = 0.002), and 4% (1%–8%, P = 0.004) for the questions regarding performing procedures for which they were not properly trained, mistakes with negative consequences to patients, and medication errors in the last year, respectively.

The most frequently sited (Likert 4 or 5) barriers to resident supervision were a lack of interest in teaching by the faculty 54.3% (99% CI 49%–59%) and lack of time due to clinical work load 47.6% (99% CI 42%–53%). Supervision scores were inversely associated with agreement to the questions regarding barriers to supervision: Lack of interest for teaching from the attending (ρ = −0.36, 95% CI −0.25 to −0.44, P < 0.001), lack of emphasis on supervision by the department leadership (ρ = −0.32, 95% CI −0.22 to −0.41, P < 0.01), lack of capability to teach from the anesthesia attending (ρ = −0.29, 95% CI −0.19 to −0.39, P < 0.01), and lack of time due to excessive clinical work (ρ = −0.18, 95% CI −0.07 to −0.28, P < 0.01).

Back to Top | Article Outline

DISCUSSION

The most important finding of the current investigation is the association between lower perceived resident supervision and the greater frequency of reported medical errors. Residents with lower supervision scores reported a higher frequency of doing procedures for which they believed not to be properly trained compared with the residents who reported higher supervision scores. More importantly, residents with lower supervision scores reported more mistakes with negative consequences to patients and more medication errors than the ones with better perceived supervision.

Another important finding of the current study is the association between trainees’ reported number of working hours and decreased perceived supervision. Trainees who reported working >70 hours per week were also more likely to report performing procedures they believed they were not properly trained. It is conceivable that anesthesiology training programs with greater clinical workloads are less able to provide adequate perceived supervision by trainees. This was further supported by the inverse relationship between perceived supervision scores and the statement that excessive clinical work was an important barrier to effective supervision. Our findings suggest that academic practices with greater clinical demand and higher resident workloads face greater challenges to provide adequate perceived supervision of trainees.

We also observed an association between residents’ year of training and lower perceived supervision. Clinical anesthesia trainees during the second year (CA-2) reported lower supervision scores than trainees during their first (CA-1) or third (CA-3) clinical anesthesia years. More importantly, CA-2 residents also reported more errors in each of the 3 questions regarding errors than residents in different clinical years. Our results are similar to the findings of Posner and Freund12 that revealed higher rates of escalation of care, operational inefficiencies, and critical incidents in CA-2 residents examined at a single institution. The investigators suggested that during the CA-2 year, the residents in their program had greater independence and were frequently required to use advanced knowledge during specialty rotations. Although these authors believed that direct supervision throughout the anesthesiology residency was cost prohibitive, they suggested that indirect supervision of residents, through expanding educational exposure and preanesthesia planning, during the CA-2 year could possibly reduce adverse events and outcomes. In contrast, Pacheco et al.13 found a higher rate of medication errors among senior residents in emergency medicine.

Other specialties have examined the relationship between trainees’ supervision and patient outcomes. Holliman et al.14 showed in an emergency room setting that adequate supervision of trainees could detect missed findings that were life-threatening to patients. Blumberg et al.15 demonstrated that internal medicine residents were more likely to adhere to high-risk patient protocols when hospital faculty were actively involved in the care. Jin et al.16 detected less resource utilization in psychiatric residents with more intensive supervision than residents with standard oversight. In a reanalysis of data collected in 1998 and 1999, Baldwin et al.7 evaluated supervision across multiple specialties and found working without adequate supervision varied widely across specialties. The amount of reported time with inadequate supervision was positively correlated to reports of medical errors, sleep deprivation, working while impaired, conflicts with medical staff, observations of others engaging in unethical conduct, increased use of alcohol, as well as medications to sleep, stay awake, and cope during residency.7 Working without adequate supervision was also negatively related to ratings of learning, time spent with attending physicians, quality of time with attending physicians, and overall satisfaction with the residency experience. Anesthesiology residents included in this study reported they worked without adequate supervision between less than once a month to at least once a month.

Few studies specific to the practice of anesthesia have addressed the role of supervision on patient outcomes. Schmidt et al.17 demonstrated that direct supervision by attending anesthesiologists was associated with a significant reduction of complication during emergency airway management performed by trainees. Epstein and Dexter18 demonstrated that lapses of 2:1 supervision occurred 35% of days and that the peak time occurred before 8 AM. They suggested that staggered starts or more anesthesiologists working at the beginning of the day would reduce supervision lapses. More studies examining the effects of trainee supervision on patient care and safety are needed.

Several studies have examined different factors affecting the occurrence of reported medical errors. Differences in cultures among countries have led to differences in medical error reporting.19 Work-related stress and burnout have been associated with an increased incidence of reported medical errors in different specialties.11,20 Specifically to anesthesiology, our group has demonstrated that burnout and depression can lead to more reported errors in anesthesiology residents.21 It is conceivable that burnout and/or depression may influence the responders’ perception of supervision. Future studies evaluating the relationship between burnout and perceived resident supervision are warranted.

Our study had several limitations and it is only valid if interpreted in this context. First, we did not assess resident supervision directly; rather, we examined residents’ self-reported perceived supervision. Second, the questionnaires were not answered in a controlled environment so we cannot exclude the presence of “distractors” affecting accuracy of responses. Third, we cannot assume cause and effect between perceived supervision and reported errors. We did not attempt to link the reported errors back to specific occurrences of inadequate supervision. We cannot exclude that responders used perceived supervision as a method of rationalization for an error occurrence. Rationalization is the reinterpretation of a moral situation, and blame shifting is one method of rationalization. Rationalization is common in hospitals and is a likely cause of underreporting of errors.22 Nor can we exclude the possibility that the lower perceived supervision scores in residents that reported greater error rates reflect the responders’ belief that they have been responsible for an event or error and this may have long-lasting emotional impact that challenges their beliefs about their educational experience or even their confidence as physicians.23 Lastly, we used self-reported errors and not actual errors as our primary outcome. Nevertheless, previous investigations suggest that physician self-reported errors may be at least as sensitive as retrospective case review.24,25

We detected an association between lower perceived supervision by anesthesiology trainees and a greater incidence of reported performance of procedures without perceived adequate training, medical errors with negative consequences to patients, and drug errors within the last year. It seems that busy academic programs and trainees during their second clinical anesthesia year (CA-2) are at higher risk. Academic programs should examine and emphasize methods to increase residents’ perceived quality of supervision in an effort to improve patient care and safety.

Back to Top | Article Outline

DISCLOSURES

Name: Gildasio S. De Oliveira, Jr., MD, MSCI.

Contribution: Study design, conduct of study, data analysis, and manuscript preparation.

Attestation: This author attests to the integrity of the data.

Name: Rod Rahmani, BS.

Contribution: Conduct of study and manuscript preparation.

Attestation: This author attests to the integrity of the data.

Name: Paul C. Fitzgerald, MS.

Contribution: Conduct of study, data analysis, and manuscript preparation.

Attestation: This author attests to the integrity of the data.

Name: Ray Chang, BS.

Contribution: Conduct of study and manuscript preparation.

Attestation: This author attests to the integrity of the data.

Name: Robert J. McCarthy, PharmD.

Contribution: Data analysis and manuscript preparation.

Attestation: This author attests to the integrity of the data.

This manuscript was handled by: Franklin Dexter, MD, PhD.

Back to Top | Article Outline

REFERENCES

1. Cottrell D, Kilminster S, Jolly B, Grant J. What is effective supervision and how does it happen? A critical incident study. Med Educ. 2002;36:1042–9

2. Bush RW. Supervision in medical education: logical fallacies and clear choices. J Grad Med Educ. 2010;2:141–3

3. Kilminster S, Cottrell D, Grant J, Jolly B. AMEE guide no. 27: effective educational and clinical supervision. Med Teach. 2007;29:2–19

4. Leach DC. Professionalism: the formation of physicians. Am J Bioeth. 2004;4:11–2

5. Maker VK, Curtis KD, Donnelly MB. Are you a surgical role model? Curr Surg. 2004;61:111–5

6. Farnan JM, Petty LA, Georgitis E, Martin S, Chiu E, Prochaska M, Arora VM. A systematic review: the effect of clinical supervision on patient and residency education outcomes. Acad Med. 2012;87:428–42

7. Baldwin DWC Jr, Daugherty SR, Ryan PM. How residents view their clinical supervision: a reanalysis of classic national survey data. J Grad Med Educ. 2010;2:37–45

8. McKee M, Black N. Does the current use of junior doctors in the United Kingdom affect the quality of medical care? Soc Sci Med. 1992;34:549–58

9. de Oliveira Filho GR, Dal Mago AJ, Garcia JH, Goldschmidt R. An instrument designed for faculty supervision evaluation by anesthesia residents and its psychometric properties. Anesth Analg. 2008;107:1316–22

10. West CP, Huschka MM, Novotny PJ, Sloan JA, Kolars JC, Habermann TM, Shanafelt TD. Association of perceived medical errors with resident distress and empathy: a prospective longitudinal study. JAMA. 2006;296:1071–8

11. Prins JT, van der Heijden FM, Hoekstra-Weebers JE, Bakker AB, van de Wiel HB, Jacobs B, Gazendam-Donofrio SM. Burnout, engagement and resident physicians’ self-reported errors. Psychol Health Med. 2009;14:654–66

12. Posner KL, Freund PR. Resident training level and quality of anesthesia care in a university hospital. Anesth Analg. 2004;98:437–42

13. Pacheco GS, Viscusi C, Hays DP, Woolridge DP. The effects of resident level of training on the rate of pediatric prescription errors in an academic emergency department. J Emerg Med. 2012;43:e343–8

14. Holliman CJ, Wuerz RC, Kimak MJ, Burkhart KK, Donovan JW, Rudnick HL, Bates MA, Muller HA. Attending supervision of nonemergency medicine residents in a university hospital ED. Am J Emerg Med. 1995;13:259–61

15. Blumberg ML, Joseph AM, Freeman JR. A strategy for improving the supervision and performance of moonlighting residents. Acad Med. 1995;70:155–7

16. Jin C, Novik S, Saravay S. Consultation-liaison psychiatry training and supervision results in fewer recommendations for constant observation. Gen Hosp Psychiatry. 2000;22:359–64

17. Schmidt UH, Kumwilaisak K, Bittner E, George E, Hess D. Effects of supervision by attending anesthesiologists on complications of emergency tracheal intubation. Anesthesiology. 2008;109:973–7

18. Epstein RH, Dexter F. Influence of supervision ratios by anesthesiologists on first-case starts and critical portions of anesthetics. Anesthesiology. 2012;116:683–91

19. Schwappach DL. Risk factors for patient-reported medical errors in eleven countries. Health Expect. 2012 Feb 1 doi: 10.1111/j.1369-7625.2011.00755.x. [Epub ahead of print]

20. Shanafelt TD, Balch CM, Bechamps G, Russell T, Dyrbye L, Satele D, Collicott P, Novotny PJ, Sloan J, Freischlag J. Burnout and medical errors among American surgeons. Ann Surg. 2010;251:995–1000

21. De Oliveira GS Jr, Fitzgerald P, Ahmad S, McCarthy RJ A national survey on the prevalence of burnout and depression among anesthesia residents: implications to patient care and safety. Available at: http://www.asabastracts.com. 2011: A027

22. Banja J, Porto G, Banja J. Medical errors and medical narcissism. Rationalization. 2005 Sudbury, MA Jones and Bartlett:19–46

23. Gazoni FM, Amato PE, Malik ZM, Durieux ME. The impact of perioperative catastrophes on anesthesiologists: results of a national survey. Anesth Analg. 2012;114:596–603

24. O’Neil AC, Petersen LA, Cook EF, Bates DW, Lee TH, Brennan TA. Physician reporting compared with medical-record review to identify adverse medical events. Ann Intern Med. 1993;119:370–6

25. Katz RI, Lagasse RS. Factors influencing the reporting of adverse perioperative outcomes to a quality management program. Anesth Analg. 2000;90:344–50

Cited By:

This article has been cited 1 time(s).

Anesthesia & Analgesia
Interpretation of the Association Between Frequency of Self-Reported Medical Errors and Faculty Supervision of Anesthesiology Residents
de Oliveira Filho, GR; Dexter, F
Anesthesia & Analgesia, 116(4): 752-753.
10.1213/ANE.0b013e3182831346
PDF (377) | CrossRef
Back to Top | Article Outline

© 2013 International Anesthesia Research Society

Login

Become a Society Member