We made minor changes to the wording (e.g., tense) of Dannefer et al.’s scale to be relevant to anesthesia providers (Table 1). Then, synonyms to terms in the scale (Table 1) were created inductively (Table 4). Examples of each use from the comments are provided in Table 5. Analyses were performed without regard to capitalization (i.e., used the words and not case). No consideration was made for spelling mistakes in either the counts of use of individual words or phrases in Table 4 or of all words combined in Table 5 (i.e., misspelled words were not counted). This approach resulted in deliberate underestimation of the percentage of words and phrases of the theme of work habits (see Limitations).
Among 41 numerical questions based on the American College of Graduate Medical Education (ACGME) competencies (Table 2 column 2),5 19.5% included questions consistent with the theme of work habits (column 3). We considered that if 20% of faculty comments were consistent with the theme, that finding would represent an important percentage. To make comparisons with 20%, we calculated lower 99.99% confidence limits on incidences. We used 99.99% to correspond with “P < 0.0001.” Also, for calculating the confidence intervals, we deliberately used the conservative Clopper-Pearson method.6,7 Calculations were performed using Microsoft Excel 2010 (Microsoft Corporation, Redmond, WA).
Multiple themes may be of concern to faculty and yet be absent from the numerical questions (Table 2) and/or written comments regarding residents because the themes are not related to daily resident clinical performance (e.g., cost of local housing). Consequently, when both numerical questions and written comments about a theme are absent, this indicates that the faculty may consider the theme unimportant with regard to resident performance. In addition, when there are a sufficient number of numerical questions to address a theme adequately, it would be expected to result in nearly 0 written comments regarding that theme. For example, if the Table 1 work habit scale had been included daily, there may have been negligible written comments about this theme; our department’s observations would then have not provided scientific value. Thus, absence of written comments cannot conclusively reveal that a theme is an unimportant feature of daily resident clinical performance, although that is quite possible (e.g., cost of housing). Absence of written comments reveals the theme is considered sufficiently (or overly) addressed by numerical questions. To evaluate this relationship, we created a control using a resident competency item that we expected to be of infrequent concern during operating room anesthesia: “patient and family conflicts.”c One numerical question regarding this competency was asked weekly (Table 2, Tuesdays, last row).
Approximately half (50.7% [lower 99.99% confidence limit, 48.4%]) of all faculty comments regarding resident performance contained the theme of work habits (Table 6). Multiple sensitivity analyses were performed excluding individual faculty, residents, and words. The lower confidence limit for comments containing the theme of work habits exceeded 42.7% under all conditions. Because each of these comments complemented 1 to 3 numerical questions including the theme (Table 2), the combination shows that the faculty anesthesiologists consider work habits to be an important theme in resident performance.
No comment included the theme of patient and family conflicts (0% [upper 99.99% confidence limit, 0.1%]).d This result suggests that, in contrast to work habits, the faculty consider 1 numerical question asked weekly to be sufficient consideration of that theme.
We evaluated whether our consideration of work habits could be just a subset of a broader theme of nonclinical performance.2 Words associated with professionalism were present in 12.8% of comments (99.99% lower limit, 11.3%)8,9: compassion (6.8%), respect (2.5%), rapport (1.5%), pleasant (1.3%), listen (0.9%), honest (0.2%), and integrity (0.1%). However, the percentage of comments including the theme of work habits and/or professionalism (i.e., nontechnical skills in general) was only 2.2% (lower limit, 1.6%) greater than for work habits with/without professionalism.
Finally, there may have been a tiny change over time in the percentage of comments that were related to the theme of work habits (Kendall’s τb = −0.024, P = 0.018). We reviewed 9 years of department-wide e-mails and faculty meeting minutes and presentations. The phrase “work habit,” singular or plural, never appeared. Furthermore, during the past 2 years, the faculty have been evaluated daily10–19 on their quality of supervision, using a valid10–14 and dependable (in a psychometric sense)11–13,15–18 scale. The quality of supervision is an independent measure of the contribution of the anesthesiologists to the care of the patient.18 As shown in Table 7, the 9 items used to evaluate faculty supervision have essentially no overlap with the 6 items used to evaluate work habits. Thus, the comments about work habits were unlikely cued from information unique to our department (i.e., our findings are likely generalizable).
Across many sensitivity analyses, an important percentage (≅50%) of comments made by faculty anesthesiologists about resident performance pertained to resident work habits. The fact that faculty took the time to make these comments shows that faculty consider (1) work habits to be an important theme in resident performance and (2) existing assessment methods in our department (e.g., numerical questions based on the ACGME competencies, Table 2) not to adequately weight or characterize this trait.
Our findings are novel. Despite the study from the American Board of Anesthesiology in 1994, there has been hardly any consideration of work habits in the specialty of anesthesiology.e Still, our findings are compatible with the report by Smith et al.,20 in which important attributes of excellence in anesthesia were characterized by personal work habits (e.g. “…critically appraises own practice,” “conscientious,” “strive[s] for perfection...,” “attention to detail,” “…problem solving,” and assumption of personal responsibility). Similarly, in the report by Larsson and Holmström,21 a structured, responsible, and focused way of approaching work tasks was considered 1 of 6 key qualities of excellent clinical performance in anesthesia. Therefore, our study and the work of others2,20,21 suggest that, provided work habits can be assessed reliably, work habits should be a component of routine resident evaluations.
The frequency of comments related to work habits has relevance to current educational goals. Descriptors of competence in anesthesia include vigilance, decisiveness, confidence, pattern recognition, flexibility, leadership, assertiveness, responsiveness, and communicativeness.22 These descriptors from Table 1 of Ref. 22, determined through a Delphi process involving Canadian anesthesiology educators, have similarities to the work habit skills adapted from the study by Dannefer et al.3 (Table 1). The ACGME has designed anesthesia-specific milestones for the assessment of residents.c These milestones are competency based and are intended to provide a template for evaluation using measurable outcomes. The work habits scale can be translated into ACGME competencies and milestones. For example, “prepared for cases,” “solv[ing] problems” and “intelligent interpretation of data,” identification and efficient completion of tasks, and “think[ing] and work[ing] independently” translate to several milestones within the patient care competency; communicating reasoning translates to milestones within the interpersonal and communications competency; and “[takes] initiative and provide[s] leadership” translates to milestones in professionalism and patient care.
Only 28% of evaluations contained a comment (Table 3). Because work habits can be improved among medical students,4 residents may already be performing at or above expectations because of earlier coaching. The overwhelming majority of comments about work habits were positive (Table 4); this is why we did not use sentiment analysis.23 On the other hand, an alternative explanation for there being mostly positive comments was that the faculty anesthesiologists might have been uncomfortable being negative (e.g., because of fear of resident retaliation when evaluating faculty performance [Table 7]). Regardless, the sensitivity analyses for faculty anesthesiologists and residents in Table 6 show that these observations did not influence the results substantively (i.e., our conclusions).
Our study was limited to evaluating the validity of monitoring the work habits of anesthesia residents. However, the items of Dannefer et al.’s work habit scale (Table 1) seem applicable to anesthesia providers in general. If the work habits scale were found (in future studies) to be reliable in routine use for anesthesia providers, then evaluating work habits would be valuable. Because of extensive activities of supervising anesthesiologists outside of operating rooms on the day of surgery (e.g., preoperative holding area and postanesthesia care unit), clinical productivity of supervising anesthesiologists is (highly) inaccurately estimated based only on the overall number of anesthetizing locations per anesthesiologist.24–27 The ability to measure the quality of supervision by anesthesiologists and the work habits of different anesthesia providers (i.e., anesthesia residents and nurse anesthetists) would facilitate measurement of the strategic results of choices in relative numbers of supervising anesthesiologists and anesthesia providers. Our study of comments of residents alone was insufficient to examine this potential.
Our article has additional limitations that each would cause the underestimation of the relevance of work habits to faculty evaluations of residents. First, we did not correct spelling errors. This reduced the number of matches of words. Second, we used a conservative method of calculating confidence limits, nonetheless used 99.99%. Third, written comments were made along with questions to be answered with radio buttons (Table 2). The questions overlapping with work habits would reduce the need to include a comment on the same theme. Fourth, over the studied period and years before, our department had not suggested to faculty that work habits are an important theme in resident activity. To the extent that these 4 conditions would reduce the generalizability of our department’s results to others, other departments’ percentages would likely be greater than ours. This reinforces our sole conclusion: work habits are a valid component of faculty anesthesiologists’ evaluations of anesthesia residents.
Name: Franklin Dexter, MD, PhD.
Contribution: This author helped design the study, analyze the data, and write the manuscript. This author is the archival author.
Attestation: Franklin Dexter has seen the original study data and approved the final manuscript.
Name: Danielle Masursky, PhD.
Contribution: This author helped conduct the study.
Attestation: Danielle Masursky has seen the original study data and approved the final manuscript.
Name: Debra Szeluga, MD, PhD.
Contribution: This author helped write the manuscript.
Attestation: Debra Szeluga has seen the original study data and approved the final manuscript.
Name: Bradley J. Hindman, MD.
Contribution: This author helped write the manuscript.
Attestation: Bradley J. Hindman has seen the original study data and approved the final manuscript.
Dr. Franklin Dexter is the Statistical Editor for Anesthesia & Analgesia. This manuscript was handled by Dr. Steven L. Shafer, Editor-in-Chief, and Dr. Dexter was not involved in any way with the editorial process or decision.
Ms. Jennifer Espy of the University of Iowa’s Department of Anesthesia created a preliminary set of synonyms for work habits and edited parts of the manuscript.
1. Bishop J. Occupation-specific versus general education and training. Ann Am Acad Polit Soc Sci. 1998;559:24–38.
2. Slogoff S, Hughes FP, Hug CC Jr, Longnecker DE, Saidman LJ. A demonstration of validity for certification by the American Board of Anesthesiology. Acad Med. 1994;69:740–6.
3. Dannefer EF, Henson LC, Bierer SB, Grady-Weliky TA, Meldrum S, Nofziger AC, Barclay C, Epstein RM. Peer assessment of professional competence. Med Educ. 2005;39:713–22.
4. Spandorfer J, Puklus T, Rose V, Vahedi M, Collins L, Giordano C, Schmidt R, Braster C. Peer assessment among first year medical students in anatomy. Anat Sci Educ. 2014;7:144–52.
5. Schartel SA, Kuhn C, Culley DJ, Wood M, Cohen N. Development of the anesthesiology educational milestones. J Grad Med Educ. 2014;6:12–4.
6. Clopper CJ, Pearson ES. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika. 1934;26:404–13.
7. Hahn GJ, Meeker WQ. Statistical Intervals. A Guide for Practitioners. 1991. New York, NY: Wiley. 82–4, 100–5.
8. McKenna J, Rosen HD. Competency-based professionalism in anesthesiology: continuing professional development. Can J Anaesth. 2012;59:889–908.
9. Meng L, Metro DG, Patel RM. Evaluating professionalism and interpersonal and communication skills: implementing a 360-degree evaluation instrument in an anesthesiology residency program. J Grad Med Educ. 2009;1:216–20.
10. De Oliveira GS Jr, Rahmani R, Fitzgerald PC, Chang R, McCarthy RJ. The association between frequency of self-reported medical errors and anesthesia trainee supervision: a survey of United States anesthesiology residents-in-training. Anesth Analg. 2013;116:892–7.
11. Hindman BJ, Dexter F, Kreiter CD, Wachtel RE. Determinants, associations, and psychometric properties of resident assessments of anesthesiologist operating room supervision. Anesth Analg. 2013;116:1342–51.
12. Dexter F, Ledolter J, Smith TC, Griffiths D, Hindman BJ. Influence of provider type (nurse anesthetist or resident physician), staff assignments, and other covariates on daily evaluations of anesthesiologists’ quality of supervision. Anesth Analg. 2014;119:670–8.
13. De Oliveira GS Jr, Dexter F, Bialek JM, McCarthy RJ. Reliability and validity of assessing subspecialty level of faculty anesthesiologists’ supervision of anesthesiology residents. Anesth Analg. 2015;120:209–13.
14. Dexter F, Hindman BJ. Quality of supervision as an independent contributor to an anesthesiologist’s individual clinical value. Anesth Analg. 2015;121:507–13.
15. de Oliveira Filho GR, Dal Mago AJ, Garcia JH, Goldschmidt R. An instrument designed for faculty supervision evaluation by anesthesia residents and its psychometric properties. Anesth Analg. 2008;107:1316–22.
16. Dexter F, Logvinov II, Brull SJ. Anesthesiology residents’ and nurse anesthetists’ perceptions of effective clinical faculty supervision by anesthesiologists. Anesth Analg. 2013;116:1352–5.
17. Hindman BJ, Dexter F, Smith TC. Anesthesia residents’ global (departmental) evaluation of faculty anesthesiologists’ supervision can be less than their average evaluations of individual anesthesiologists. Anesth Analg. 2015;120:204–8.
18. Dexter F, Masursky D, Hindman BJ. Reliability and validity of the anesthesiologist supervision instrument when certified registered nurse anesthetists provide scores. Anesth Analg. 2015;120:214–9.
19. Dexter F, Ledolter J, Hindman BJ. Bernoulli Cumulative Sum (CUSUM) control charts for monitoring of anesthesiologists’ performance in supervising anesthesia residents and nurse anesthetists. Anesth Analg. 2014;119:679–85.
20. Smith AF, Glavin R, Greaves JD. Defining excellence in anaesthesia: the role of personal qualities and practice environment. Br J Anaesth. 2011;106:38–43.
21. Larsson J, Holmström IK. How excellent anaesthetists perform in the operating theatre: a qualitative study on non-technical skills. Br J Anaesth. 2013;110:115–21.
22. Kearney RA. Defining professionalism in anaesthesiology. Med Educ. 2005;39:769–76.
23. Rhoton MF. A new method to evaluate clinical performance and critical incidence in anaesthesia: quantification of daily comments by teachers. Med Educ. 1989;23:280–9.
24. Epstein RH, Dexter F. Influence of supervision ratios by anesthesiologists on first-case starts and critical portions of anesthetics. Anesthesiology. 2012;116:683–91.
25. Smallman B, Dexter F, Masursky D, Li F, Gorji R, George D, Epstein RH. Role of communication systems in coordinating supervising anesthesiologists’ activities outside of operating rooms. Anesth Analg. 2013;116:898–903.
26. Epstein RH, Dexter F, Lopez MG, Ehrenfeld JM. Anesthesiologist staffing considerations consequent to the temporal distribution of hypoxemic episodes in the postanesthesia care unit. Anesth Analg. 2014;119:1322–33.
Copyright © 2016 International Anesthesia Research Society
27. Dexter F, Wachtel RE, Todd MM, Hindman BJ. The “Fourth Mission”: the time commitment of anesthesiology faculty for management is comparable to their time commitments to education, research, and indirect patient care. A A Case Rep. 2015;5:206–11.