Share this article on:

Work Habits Are Valid Components of Evaluations of Anesthesia Residents Based on Faculty Anesthesiologists’ Daily Written Comments About Residents

Dexter, Franklin MD, PhD; Masursky, Danielle PhD; Szeluga, Debra MD, PhD; Hindman, Bradley J. MD

doi: 10.1213/ANE.0000000000001199
Economics, Education, and Policy: Research Report

BACKGROUND: In our department, faculty anesthesiologists routinely evaluate the resident physicians with whom they worked in an operative setting the day before, providing numerical scores to questions. The faculty can also enter a written comment if so desired. Because residents’ work habits are important to anesthesiology program directors, and work habits can improve with feedback, we hypothesized that faculty comments would include the theme of the anesthesia resident’s work habits.

METHODS: We analyzed all 6692 faculty comments from January 1, 2011, to June 30, 2015. We quantified use of the theme of Dannefer et al.’s work habit scale, specifically the words and phrases in the scale, and synonyms to the words.

RESULTS: Approximately half (50.7% [lower 99.99% confidence limit, 48.4%]) of faculty comments contained the theme of work habits. Multiple sensitivity analyses were performed excluding individual faculty, residents, and words. The lower confidence limits for comments containing the theme were each >42.7%.

CONCLUSIONS: Although faculty anesthesiologists completed (numerical) questions based on the American College of Graduate Medical Education competencies to evaluate residents, an important percentage of written comments included the theme of work habits. The implication is that the theme has validity as one component of the routine evaluation of anesthesia residents.

Published ahead of print March 9, 2016

From the *Division of Management Consulting, Department of Anesthesia, University of Iowa, Iowa City, Iowa; Active Aging and Community Engagement Center, State University of New York at Oswego, Oswego, New York; and Department of Anesthesia, University of Iowa, Iowa City, Iowa.

Accepted for publication January 4, 2016.

Published ahead of print March 9, 2016

Funding: Departmental.

The authors declare no conflicts of interest.

An abstract describing this work has been submitted to the Association of University Anesthesiologists meeting in San Francisco, California, May 2016.

Reprints will not be available from the authors.

Address correspondence to Franklin Dexter, MD, PhD, Division of Management Consulting, Department of Anesthesia, University of Iowa, 200 Hawkins Dr., 6JCP, Iowa City, IA 52242. Address e-mail to Franklin-Dexter@UIowa.edu or www.FranklinDexter.net.

“Owners of small- and medium-sized businesses” in the United States of America “were asked” “which abilities influence hiring selections the most?”1 “The trait that most directly rivaled occupational skills (i.e., the ability to do the job) was ‘work habits.’”1 “Work habits and attitude (trying hard, enthusiasm, punctuality)” were ranked as the first or second most important trait by 65% of employers.1 “Occupational [and] job skills” was ranked first or second by 54% of employers. “One year later,” “the two” traits “with the largest” associations with productivity were “the ability to learn new occupational and job skills” and “ex-post assessments of work habits.”1

Similarly, anesthesia residents’ work habits are of importance to anesthesiology program directors. Program directors were asked to specify the number of surgical procedures for which each director “would permit” each of their graduating residents “to provide anesthesia care for” them personally: “elective cholecystectomy,” “laparotomy for acute bowel obstruction,” and/or “sitting posterior fossa craniotomy.”2 There was a strong association between the number of allowed surgical procedures (0–3) and the resident passing both the written and the oral American Board of Anesthesiology examinations on the first attempt.2 Program directors’ assessments of each resident’s work habits of “responsibility,” “reliability and punctuality,” “industriousness,” “motivation,” and “adaptability” were each strongly and significantly associated with the number of allowed procedures.2 Thus, assessments of the work habits of anesthesia residents are linked with assessments of their clinical competency and the latter with board certification.

The work habit scale of Dannefer et al.3 contains 6 items (Table 1). The scale was developed to assess medical students.3 The unidimensional instrument has large internal consistency (Cronbach α = 0.94).3 Furthermore, with feedback, work habits improve.4 Specifically, in the middle of an anatomy course, first-year medical students received online peer feedback using Dannefer et al.’s scale.4 Final assessments showed significant improvements (e.g., in taking “initiative and providing leadership”).4 There also were significant correlations between course grades and 3 prefeedback and postfeedback work habits: consistently well prepared, clearly communicated his or her reasoning process, and leadership qualities.4 Thus, trainee (medical student) work habits are important, and feedback regarding work habits can result in favorable changes in behaviors.

Table 1

Table 1

In the current study, we describe the results of a textual content analysis of faculty anesthesiologists’ written comments about anesthesia residents. Because work habits (Table 1) are important to employers in general,1 because residents’ work habits are important to anesthesiology program directors,2 and because work habits can improve with feedback,4 we hypothesized that faculty comments would include the theme of the anesthesia resident’s work habits.

Back to Top | Article Outline

METHODS

The University of Iowa IRB declared that this investigation did not meet the regulatory definition of human subjects research.a All work was performed with deidentified data.

Faculty anesthesiologists at the University of Iowa are asked routinely to evaluate the residents with whom they worked the previous day in an operating room setting, including obstetrics and/or non-operating room anesthesia (e.g., radiation therapy). Faculty input numerical scores using radio buttons to answer a rotating set of questions (Table 2).b,5 The faculty can also enter a written comment regarding resident performance if they so desire (i.e., 0 comments or 1 comment per day; Table 3).

Table 2

Table 2

Table 3

Table 3

We analyzed every written comment from the first date of this system (January 1, 2011) through the last day of the most recently completed residency class (June 30, 2015). There were 6692 comments among the clinical anesthesia year 1 to 3 residents (Table 3). Our aim was to estimate the percentage of all comments that were related to the theme of work habits (Table 4).

Table 4

Table 4

Table 5

Table 5

We made minor changes to the wording (e.g., tense) of Dannefer et al.’s scale to be relevant to anesthesia providers (Table 1). Then, synonyms to terms in the scale (Table 1) were created inductively (Table 4). Examples of each use from the comments are provided in Table 5. Analyses were performed without regard to capitalization (i.e., used the words and not case). No consideration was made for spelling mistakes in either the counts of use of individual words or phrases in Table 4 or of all words combined in Table 5 (i.e., misspelled words were not counted). This approach resulted in deliberate underestimation of the percentage of words and phrases of the theme of work habits (see Limitations).

Back to Top | Article Outline

Statistical Methods

Among 41 numerical questions based on the American College of Graduate Medical Education (ACGME) competencies (Table 2 column 2),5 19.5% included questions consistent with the theme of work habits (column 3). We considered that if 20% of faculty comments were consistent with the theme, that finding would represent an important percentage. To make comparisons with 20%, we calculated lower 99.99% confidence limits on incidences. We used 99.99% to correspond with “P < 0.0001.” Also, for calculating the confidence intervals, we deliberately used the conservative Clopper-Pearson method.6,7 Calculations were performed using Microsoft Excel 2010 (Microsoft Corporation, Redmond, WA).

Multiple themes may be of concern to faculty and yet be absent from the numerical questions (Table 2) and/or written comments regarding residents because the themes are not related to daily resident clinical performance (e.g., cost of local housing). Consequently, when both numerical questions and written comments about a theme are absent, this indicates that the faculty may consider the theme unimportant with regard to resident performance. In addition, when there are a sufficient number of numerical questions to address a theme adequately, it would be expected to result in nearly 0 written comments regarding that theme. For example, if the Table 1 work habit scale had been included daily, there may have been negligible written comments about this theme; our department’s observations would then have not provided scientific value. Thus, absence of written comments cannot conclusively reveal that a theme is an unimportant feature of daily resident clinical performance, although that is quite possible (e.g., cost of housing). Absence of written comments reveals the theme is considered sufficiently (or overly) addressed by numerical questions. To evaluate this relationship, we created a control using a resident competency item that we expected to be of infrequent concern during operating room anesthesia: “patient and family conflicts.”c One numerical question regarding this competency was asked weekly (Table 2, Tuesdays, last row).

Back to Top | Article Outline

RESULTS

Approximately half (50.7% [lower 99.99% confidence limit, 48.4%]) of all faculty comments regarding resident performance contained the theme of work habits (Table 6). Multiple sensitivity analyses were performed excluding individual faculty, residents, and words. The lower confidence limit for comments containing the theme of work habits exceeded 42.7% under all conditions. Because each of these comments complemented 1 to 3 numerical questions including the theme (Table 2), the combination shows that the faculty anesthesiologists consider work habits to be an important theme in resident performance.

Table 6

Table 6

No comment included the theme of patient and family conflicts (0% [upper 99.99% confidence limit, 0.1%]).d This result suggests that, in contrast to work habits, the faculty consider 1 numerical question asked weekly to be sufficient consideration of that theme.

We evaluated whether our consideration of work habits could be just a subset of a broader theme of nonclinical performance.2 Words associated with professionalism were present in 12.8% of comments (99.99% lower limit, 11.3%)8,9: compassion (6.8%), respect (2.5%), rapport (1.5%), pleasant (1.3%), listen (0.9%), honest (0.2%), and integrity (0.1%). However, the percentage of comments including the theme of work habits and/or professionalism (i.e., nontechnical skills in general) was only 2.2% (lower limit, 1.6%) greater than for work habits with/without professionalism.

Table 7

Table 7

Finally, there may have been a tiny change over time in the percentage of comments that were related to the theme of work habits (Kendall’s τb = −0.024, P = 0.018). We reviewed 9 years of department-wide e-mails and faculty meeting minutes and presentations. The phrase “work habit,” singular or plural, never appeared. Furthermore, during the past 2 years, the faculty have been evaluated daily10–19 on their quality of supervision, using a valid10–14 and dependable (in a psychometric sense)11–13,15–18 scale. The quality of supervision is an independent measure of the contribution of the anesthesiologists to the care of the patient.18 As shown in Table 7, the 9 items used to evaluate faculty supervision have essentially no overlap with the 6 items used to evaluate work habits. Thus, the comments about work habits were unlikely cued from information unique to our department (i.e., our findings are likely generalizable).

Back to Top | Article Outline

DISCUSSION

Across many sensitivity analyses, an important percentage (≅50%) of comments made by faculty anesthesiologists about resident performance pertained to resident work habits. The fact that faculty took the time to make these comments shows that faculty consider (1) work habits to be an important theme in resident performance and (2) existing assessment methods in our department (e.g., numerical questions based on the ACGME competencies, Table 2) not to adequately weight or characterize this trait.

Our findings are novel. Despite the study from the American Board of Anesthesiology in 1994, there has been hardly any consideration of work habits in the specialty of anesthesiology.e Still, our findings are compatible with the report by Smith et al.,20 in which important attributes of excellence in anesthesia were characterized by personal work habits (e.g. “…critically appraises own practice,” “conscientious,” “strive[s] for perfection...,” “attention to detail,” “…problem solving,” and assumption of personal responsibility). Similarly, in the report by Larsson and Holmström,21 a structured, responsible, and focused way of approaching work tasks was considered 1 of 6 key qualities of excellent clinical performance in anesthesia. Therefore, our study and the work of others2,20,21 suggest that, provided work habits can be assessed reliably, work habits should be a component of routine resident evaluations.

The frequency of comments related to work habits has relevance to current educational goals. Descriptors of competence in anesthesia include vigilance, decisiveness, confidence, pattern recognition, flexibility, leadership, assertiveness, responsiveness, and communicativeness.22 These descriptors from Table 1 of Ref. 22, determined through a Delphi process involving Canadian anesthesiology educators, have similarities to the work habit skills adapted from the study by Dannefer et al.3 (Table 1). The ACGME has designed anesthesia-specific milestones for the assessment of residents.c These milestones are competency based and are intended to provide a template for evaluation using measurable outcomes. The work habits scale can be translated into ACGME competencies and milestones. For example, “prepared for cases,” “solv[ing] problems” and “intelligent interpretation of data,” identification and efficient completion of tasks, and “think[ing] and work[ing] independently” translate to several milestones within the patient care competency; communicating reasoning translates to milestones within the interpersonal and communications competency; and “[takes] initiative and provide[s] leadership” translates to milestones in professionalism and patient care.

Back to Top | Article Outline

Limitations

Only 28% of evaluations contained a comment (Table 3). Because work habits can be improved among medical students,4 residents may already be performing at or above expectations because of earlier coaching. The overwhelming majority of comments about work habits were positive (Table 4); this is why we did not use sentiment analysis.23 On the other hand, an alternative explanation for there being mostly positive comments was that the faculty anesthesiologists might have been uncomfortable being negative (e.g., because of fear of resident retaliation when evaluating faculty performance [Table 7]). Regardless, the sensitivity analyses for faculty anesthesiologists and residents in Table 6 show that these observations did not influence the results substantively (i.e., our conclusions).

Our study was limited to evaluating the validity of monitoring the work habits of anesthesia residents. However, the items of Dannefer et al.’s work habit scale (Table 1) seem applicable to anesthesia providers in general. If the work habits scale were found (in future studies) to be reliable in routine use for anesthesia providers, then evaluating work habits would be valuable. Because of extensive activities of supervising anesthesiologists outside of operating rooms on the day of surgery (e.g., preoperative holding area and postanesthesia care unit), clinical productivity of supervising anesthesiologists is (highly) inaccurately estimated based only on the overall number of anesthetizing locations per anesthesiologist.24–27 The ability to measure the quality of supervision by anesthesiologists and the work habits of different anesthesia providers (i.e., anesthesia residents and nurse anesthetists) would facilitate measurement of the strategic results of choices in relative numbers of supervising anesthesiologists and anesthesia providers. Our study of comments of residents alone was insufficient to examine this potential.

Our article has additional limitations that each would cause the underestimation of the relevance of work habits to faculty evaluations of residents. First, we did not correct spelling errors. This reduced the number of matches of words. Second, we used a conservative method of calculating confidence limits, nonetheless used 99.99%. Third, written comments were made along with questions to be answered with radio buttons (Table 2). The questions overlapping with work habits would reduce the need to include a comment on the same theme. Fourth, over the studied period and years before, our department had not suggested to faculty that work habits are an important theme in resident activity. To the extent that these 4 conditions would reduce the generalizability of our department’s results to others, other departments’ percentages would likely be greater than ours. This reinforces our sole conclusion: work habits are a valid component of faculty anesthesiologists’ evaluations of anesthesia residents.

Back to Top | Article Outline

DISCLOSURES

Name: Franklin Dexter, MD, PhD.

Contribution: This author helped design the study, analyze the data, and write the manuscript. This author is the archival author.

Attestation: Franklin Dexter has seen the original study data and approved the final manuscript.

Name: Danielle Masursky, PhD.

Contribution: This author helped conduct the study.

Attestation: Danielle Masursky has seen the original study data and approved the final manuscript.

Name: Debra Szeluga, MD, PhD.

Contribution: This author helped write the manuscript.

Attestation: Debra Szeluga has seen the original study data and approved the final manuscript.

Name: Bradley J. Hindman, MD.

Contribution: This author helped write the manuscript.

Attestation: Bradley J. Hindman has seen the original study data and approved the final manuscript.

Back to Top | Article Outline

RECUSE NOTE

Dr. Franklin Dexter is the Statistical Editor for Anesthesia & Analgesia. This manuscript was handled by Dr. Steven L. Shafer, Editor-in-Chief, and Dr. Dexter was not involved in any way with the editorial process or decision.

Back to Top | Article Outline

ACKNOWLEDGMENTS

Ms. Jennifer Espy of the University of Iowa’s Department of Anesthesia created a preliminary set of synonyms for work habits and edited parts of the manuscript.

Back to Top | Article Outline

FOOTNOTES

a http://FDshort.com/OHRP2008, accessed October 9, 2015; http://FDshort.com/NIH-HSR, accessed October 9, 2015; http://FDshort.com/FredHutch-HSR. Accessed October 9, 2015.
Cited Here...

b https://en.wikipedia.org/wiki/Radio_button. Accessed October 9, 2015.
Cited Here...

c http://FDshort.com/ACGMEmilestones. Accessed October 8, 2015, page 22.
Cited Here...

d Among the 6692 comments, the 3 uses of “conflict,” 2 uses of “disagree,” and 1 use of “argues” were unrelated to the theme. There were 0 uses of “argument,” “at odds,” “clash,” “dispute,” “divergent,” “fight,” or “quarrel.” These searches were performed without regard to tense (e.g., “argument” includes “arguments” and “argumentative”).
Cited Here...

e (1) By PubMed search on December 23, 2015, there were 111,451 articles from the following “[Journal]”: “aana j,” “acta anaesthesiologica scandinavica,” “anaesthesia and intensive care,” “anaesthesia,” “anesthesia and analgesia,” “anesthesiology,” “british journal of anaesthesia,” “Can J Anaesth,” “european journal of anaesthesiology,” OR “journal of clinical anesthesia.” They included not a single one (i.e., zero) of the 196 articles with “work habit”[All Fields] OR “work habits”[All Fields]. (2) Using Google Scholar, we performed full-text search of the 23 articles citing Slogoff et al.2 Not a single article (again, zero) included the phrase “work habit” OR “work habits”. (3) Using Google Scholar, we searched the full text of the 17,300 results for “work habit” OR “work habits.” The only article also including “Slogoff” was their 1994 paper.
Cited Here...

Back to Top | Article Outline

REFERENCES

1. Bishop J. Occupation-specific versus general education and training. Ann Am Acad Polit Soc Sci. 1998;559:24–38
2. Slogoff S, Hughes FP, Hug CC Jr, Longnecker DE, Saidman LJ. A demonstration of validity for certification by the American Board of Anesthesiology. Acad Med. 1994;69:740–6
3. Dannefer EF, Henson LC, Bierer SB, Grady-Weliky TA, Meldrum S, Nofziger AC, Barclay C, Epstein RM. Peer assessment of professional competence. Med Educ. 2005;39:713–22
4. Spandorfer J, Puklus T, Rose V, Vahedi M, Collins L, Giordano C, Schmidt R, Braster C. Peer assessment among first year medical students in anatomy. Anat Sci Educ. 2014;7:144–52
5. Schartel SA, Kuhn C, Culley DJ, Wood M, Cohen N. Development of the anesthesiology educational milestones. J Grad Med Educ. 2014;6:12–4
6. Clopper CJ, Pearson ES. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika. 1934;26:404–13
7. Hahn GJ, Meeker WQ Statistical Intervals. A Guide for Practitioners. 1991 New York, NY Wiley:82–4, 100–5
8. McKenna J, Rosen HD. Competency-based professionalism in anesthesiology: continuing professional development. Can J Anaesth. 2012;59:889–908
9. Meng L, Metro DG, Patel RM. Evaluating professionalism and interpersonal and communication skills: implementing a 360-degree evaluation instrument in an anesthesiology residency program. J Grad Med Educ. 2009;1:216–20
10. De Oliveira GS Jr, Rahmani R, Fitzgerald PC, Chang R, McCarthy RJ. The association between frequency of self-reported medical errors and anesthesia trainee supervision: a survey of United States anesthesiology residents-in-training. Anesth Analg. 2013;116:892–7
11. Hindman BJ, Dexter F, Kreiter CD, Wachtel RE. Determinants, associations, and psychometric properties of resident assessments of anesthesiologist operating room supervision. Anesth Analg. 2013;116:1342–51
12. Dexter F, Ledolter J, Smith TC, Griffiths D, Hindman BJ. Influence of provider type (nurse anesthetist or resident physician), staff assignments, and other covariates on daily evaluations of anesthesiologists’ quality of supervision. Anesth Analg. 2014;119:670–8
13. De Oliveira GS Jr, Dexter F, Bialek JM, McCarthy RJ. Reliability and validity of assessing subspecialty level of faculty anesthesiologists’ supervision of anesthesiology residents. Anesth Analg. 2015;120:209–13
14. Dexter F, Hindman BJ. Quality of supervision as an independent contributor to an anesthesiologist’s individual clinical value. Anesth Analg. 2015;121:507–13
15. de Oliveira Filho GR, Dal Mago AJ, Garcia JH, Goldschmidt R. An instrument designed for faculty supervision evaluation by anesthesia residents and its psychometric properties. Anesth Analg. 2008;107:1316–22
16. Dexter F, Logvinov II, Brull SJ. Anesthesiology residents’ and nurse anesthetists’ perceptions of effective clinical faculty supervision by anesthesiologists. Anesth Analg. 2013;116:1352–5
17. Hindman BJ, Dexter F, Smith TC. Anesthesia residents’ global (departmental) evaluation of faculty anesthesiologists’ supervision can be less than their average evaluations of individual anesthesiologists. Anesth Analg. 2015;120:204–8
18. Dexter F, Masursky D, Hindman BJ. Reliability and validity of the anesthesiologist supervision instrument when certified registered nurse anesthetists provide scores. Anesth Analg. 2015;120:214–9
19. Dexter F, Ledolter J, Hindman BJ. Bernoulli Cumulative Sum (CUSUM) control charts for monitoring of anesthesiologists’ performance in supervising anesthesia residents and nurse anesthetists. Anesth Analg. 2014;119:679–85
20. Smith AF, Glavin R, Greaves JD. Defining excellence in anaesthesia: the role of personal qualities and practice environment. Br J Anaesth. 2011;106:38–43
21. Larsson J, Holmström IK. How excellent anaesthetists perform in the operating theatre: a qualitative study on non-technical skills. Br J Anaesth. 2013;110:115–21
22. Kearney RA. Defining professionalism in anaesthesiology. Med Educ. 2005;39:769–76
23. Rhoton MF. A new method to evaluate clinical performance and critical incidence in anaesthesia: quantification of daily comments by teachers. Med Educ. 1989;23:280–9
24. Epstein RH, Dexter F. Influence of supervision ratios by anesthesiologists on first-case starts and critical portions of anesthetics. Anesthesiology. 2012;116:683–91
25. Smallman B, Dexter F, Masursky D, Li F, Gorji R, George D, Epstein RH. Role of communication systems in coordinating supervising anesthesiologists’ activities outside of operating rooms. Anesth Analg. 2013;116:898–903
26. Epstein RH, Dexter F, Lopez MG, Ehrenfeld JM. Anesthesiologist staffing considerations consequent to the temporal distribution of hypoxemic episodes in the postanesthesia care unit. Anesth Analg. 2014;119:1322–33
27. Dexter F, Wachtel RE, Todd MM, Hindman BJ. The “Fourth Mission”: the time commitment of anesthesiology faculty for management is comparable to their time commitments to education, research, and indirect patient care. A A Case Rep. 2015;5:206–11
© 2016 International Anesthesia Research Society