Secondary Logo

Journal Logo

Economics, Education, and Policy: Research Report

Academic Productivity of Directors of ACGME-Accredited Residency Programs in Surgery and Anesthesiology

Culley, Deborah J. MD*; Fahy, Brenda G. MD; Xie, Zhongcong MD, PhD; Lekowski, Robert MD*; Buetler, Sascha MD, PhD*; Liu, Xiaoxia MS*; Cohen, Neal H. MD; Crosby, Gregory MD

Author Information
doi: 10.1213/ANE.0b013e3182a8fab5
  • Free

Training of resident physicians is a challenging, multifaceted enterprise that in the course of a few years transforms a trainee from an inexperienced neophyte to a knowledgeable, skillful, independent medical practitioner and professional. This requires that the trainee acquire clinical knowledge and skills, develop sound clinical judgment, and become proficient with critical evaluation of new medical information. The responsibility for ensuring that trainees receive the requisite clinical experience and educational opportunities to become independent practitioners and lifelong learners falls to the residency program director. This is a complicated job with multiple expectations and responsibilities. One of these, as codified by the Accreditation Council for Graduate Medical Education (ACGME), which sets standards for accrediting residency programs that are then operationalized as program requirements by the various Residency Review Committees (RRCs), is participation in scholarly activities. This expectation is intended to generate in program directors an aptitude for inquiry and emphasize for residents and faculty the critical importance of questioning traditional practices, generating, disseminating, and incorporating new evidence into clinical practice, and to underscore the commitment all physicians should have to lifelong learning.

Thus, we found it interesting that the March, 2012 Newsletter of the Anesthesiology RRC reported that 16% of anesthesiology programs reviewed over the previous 2 years were cited for failure to document scholarly activity, whereas according to the Surgery RRC Newsletter from August 2011, such citations were issued to fewer than 5.9% of surgery programs. We hypothesized that this difference in raw percentage citation rate may in part reflect a discrepancy in scholarly activity between program directors of anesthesiology and surgery. To test this hypothesis, we examined scholarly productivity of current program directors of ACGME-accredited anesthesiology residency programs based on peer-reviewed publications cited in PubMed and success in obtaining funding for research from the National Institutes of Health (NIH). To control for institutional differences and benchmark anesthesiology program directors against another specialty, we performed the same analysis for program directors in surgery within the same institutions.

METHODS

Data publicly available on the Web sites of the ACGME (http://www.acgme.org), the NIH Research Portfolio Online Reporting Tools (RePORT, http://projectreporter.nih.gov/reporter.cfm), PubMed (http://www.ncbi.nlm.nih.gov/pubmed/), and the various department or medical school Web sites were surveyed. Furthermore, we queried the publicly available databases of the American Board of Surgery (ABS; www.absurgery.org) and the American Board of Anesthesiology (ABA; www.theaba.org) to determine the year of American Board of Medical Specialties (ABMS) member board certification, which we used as a surrogate for years each program director has been practicing in their specialty.

On November 1, 2011, a list of current program directors of ACGME-accredited anesthesiology and surgery residency programs was obtained from the ACGME Web site. To control for institutional differences, we only included institutions that had ACGME-accredited programs in both anesthesiology and surgery. The name of the program director and year they were first appointed to the position were obtained from the ACGME Web site. Year of board certification, verification of middle initial, and academic rank of each program director was obtained from the Web site appropriate to each specialty (i.e. the ABS or ABA) and/or, when available, applicable departmental or medical school Web sites.

The name of each individual was then entered into PubMed, and all citations to work published between 1962 and October 31, 2011 documented. Each search result was manually reviewed by 1 author for accuracy and plausibility (nature of the research; year it was performed; institution; etc.), and by a second when there was ambiguity or uncertainty. In the initial analysis, we did not differentiate between the types of publications; letters to the editor were counted the same as original research, editorials, or reviews. We did differentiate between publications related to education, defined as those containing the words “medical student,” “resident,” “fellow,” or “education” in the article title or abstract, and those that were not. Last, the name of each individual was entered into NIH RePORT, which lists grants funded by NIH between 1988 and the present. RePORT lists only the principal investigators; therefore, a person having a lesser role on a grant or listed as a coinvestigator does not appear and would not be credited.

To further evaluate scholarship and publication quality, we randomly selected 25 institutions (20% of total) and performed a post hoc analysis on this subset of institution-matched surgery and anesthesiology program directors to compare authorship position (first, middle, or last), type of publication (original research; editorials and reviews; case reports and letters to the editor), and h-index, a bibliometric statistic used widely in academic medicine to evaluate the relative quality of a person’s collective body of work.1–3 The h-index is determined mainly by how often a publication is cited and is defined as the number (h) of an investigator’s publications that have been cited at least h times.4

Statistical Analysis

The baseline characteristics of our study cohort are presented as percentages for categorical variables and mean (± SD) or median with interquartile range (lack normality data) for continuous variables. Univariate associations of the paired samples were evaluated with nonparametric methods using Wilcoxon signed rank sum test for continuous variables and McNemar test (for dichotomous data) or Stuart-Maxwell test (generalized McNemar), as appropriate, for categorical variables. Multivariate regression models with generalized estimating equations (GEE modeling) were used to examine predictors of publications since our observations are matched by institution and thus correlated. Modified Poisson regression with robust variance estimates5 was used to directly estimate rate ratios, and Spearman correlations were calculated for the multiple end points. All statistical tests were 2-sided, with a type I error of 0.05. A P-value of <0.05 was considered to be statistically significant. All statistical analyses were performed with SAS version 9.3 (SAS Institute, Cary, NC.)

RESULTS

There were 131 anesthesiology and 250 surgery ACGME-accredited residency programs as of November 1, 2011. Of these, 127 institutions had ACGME-accredited residency programs in both anesthesiology and surgery; only these programs were included in this analysis. One anesthesiology and 1 surgery program director could not be identified reliably in Pubmed because there was >1 person in that specialty publishing under the same name; these individuals and the corresponding program director in the same institution were excluded from the PubMed analysis.

Univariate analysis identified no baseline differences between the groups in terms of sex (P = 0.24), academic rank (P = 0.38), year of program director board certification (P = 0.42), number of years since board certification (P = 0.40), year of program director appointment (P = 0.22), or time serving as a program director (P = 0.15) (Table 1). There also was no difference between the specialties in the current program accreditation cycle length (P = 0.55; Table 1). However, program directors in anesthesiology had less prior or active NIH funding compared with program directors of surgical residencies (P = 0.002; Table 1) and fewer total PubMed citations (P < 0.001), education-related publications (P < 0.001), and recent publications (November 2006–October 2011; P < 0.001)(Table 2). Curiously, approximately 75% and 40% of anesthesiology and surgery program directors, respectively, had no education-related publications (Table 2). Multivariate analysis revealed that the publication rate for anesthesiology program directors was 43% (95% confidence interval, 31%–58%) of that of the corresponding program directors of surgical residency programs holding other variables constant.

Table 1
Table 1:
Baseline Characteristics
Table 2
Table 2:
Publication Comparison

Among the randomly selected subset of program directors, post hoc analysis revealed that anesthesiology program directors had fewer first (P = 0.003), middle (P = 0.002), or last author publications (P = 0.009) and fewer original manuscripts (P < 0.001) and editorials and reviews (P < 0.001) compared with program directors in surgery but similar numbers of case reports and letters to the editor (P = 0.45) (Table 3). The h-index for surgical program directors was also significantly higher than that of anesthesiology program directors (P < 0.001, Table 3).

Table 3
Table 3:
Order of Authorships and Type of Manuscript

Finally, total number of citations in PubMed, publication type, h-index, number of education-related publications, history of NIH funding, and publications at each academic rank are all highly correlated (P < 0.0001 for all correlations), with program directors of anesthesiology residency programs being lower than program directors of surgery.

DISCUSSION

This is the first study to examine and compare the scholarship credentials of program directors of ACGME-accredited anesthesiology and surgery residency programs. Whether judged by total number of citations in PubMed, publication type, h-index, number of education-related publications, history of NIH funding, or PubMed-citable publications at each academic rank, program directors of anesthesiology residency programs underperform those of surgery. The differences are sizeable; program directors for anesthesiology contribute about 40% as many publications as their surgical peers within the same institutions. This is unlikely to be due to differences in the ACGME core program requirements between the fields because in both specialties academic achievement is stipulated as a key qualification of the program director. Nor can it be explained by the ever escalating burden of monitoring and complying with onerous administrative and regulatory requirements associated with directing an ACGME-accredited residency program, as these are the same for both specialties. In addition, the ACGME has specific requirements to ensure that program directors have protected time to fulfill the responsibilities and expectations of the position. Differences in the experience or seniority of program directors are also not responsible as years since board certification or tenure as a program director was the same in both groups. Therefore, although a causal relationship with the program director’s performance cannot be established, the fact that ACGME-accredited residency programs in anesthesiology are cited more frequently than surgical programs for inadequate scholarly activity seems justified.

The ACGME and RRC require program directors to have “significant academic achievements” in their specialty. The definition of academic achievements is broad enough to allow program directors to participate in bench or clinical trials, pursue and publish creative approaches to education, or assess and report educational outcomes. This is appropriate given the service of program directors as educational leaders within their respective institutions, their influence over department educational initiatives, responsibilities for oversight of the core program, as well as all subspecialty ACGME-accredited fellowship programs, and major role in establishing and nurturing an environment of inquiry and scholarship. The latter is especially vital for trainees and the program as a whole, as a willingness to challenge assumptions and practices is crucial for lifelong learning. As such, the program director and faculty should continuously evaluate established local educational practices and newly introduced initiatives, apply and test evolving theories of postgraduate medical education, and refine them as needed to ensure training of the residents and fellows is appropriate. If done thoughtfully and systematically, the results of such assessments can and should be disseminated broadly so as to advance the specialty. The data presented here raise serious questions about the qualifications, interest, or commitment of many anesthesiology program directors to fulfill this particular responsibility.

We recognize that the nature of scholarly achievements expected of a program director may be broader than the sources of data used for this analysis. The NIH and PubMed databases used here include only NIH-funded projects and manuscripts published in peer-reviewed journals cataloged by PubMed. For academic anesthesiologists and surgeons, lectures, curriculum development, and educational leadership are important aspects of creative activities assessed by the RRCs. The databases used in this analysis do not capture such activities, but they do include the most significant scholarly activity that undergoes peer evaluation. The RRCs of both specialties consider peer-reviewed publications to be important. The Surgery RRC takes the position that the program director “sets the tone for the scholarly environment of the residency program” and, “because it is expected that both faculty members and residents are involved in research and publications, the program director should have evidence of peer-reviewed publication…” and be recognized and respected as having demonstrated success in scholarship (see 2012 Frequently Asked Questions; http://www.acgme.org/acgmeweb/ProgramandInstitutionalGuidelines/SurgicalAccreditation/Surgery.aspx). Likewise, the Anesthesiology RRC position is that participation in lectures, journal clubs, or anesthesia committees … “cannot substitute for publication in scholarly journals and other academic pursuits” (see 2012 Frequently Asked Questions; http://www.acgme.org/acgmeweb/ProgramandInstitutionalGuidelines/Hospital-BasedAccreditation/Anesthesiology.aspx). Both specialty RRCs point out that scholarship on the part of the program director and faculty is essential to their ability to influence resident scholarly activity, expose trainees to research methodology and ways to assess the literature, and, most importantly, to advance the specialty. Based on the findings of this study, anesthesiology is clearly lagging behind its surgical peers in this important dimension. However, this does not imply that the program director alone is responsible for meeting a department’s ACGME-mandated benchmarks for scholarship.

In fact, the entire faculty share responsibility for creating and encouraging a scholarly environment, but here too, anesthesiology is weak. Based on 2006 data, departments of anesthesiology receive <1% of the research dollars awarded by the NIH.6 Furthermore, a recent study of 24 accredited residency programs in the United States showed that the median number of career publications of academic anesthesiologists was just 3, with assistant, associate, and full professors having 1, 13, and 46, respectively, although the range within each rank was large.1 Similar but modestly higher numbers were reported for groups of pediatric anesthesiologists3 and academic anesthesiologists3 in Canada. None of these studies included a nonanesthesiologist control group, but similar analyses of radiation oncologists and faculty in neurosurgery and urology suggest anesthesiologists publish less often than do these other specialists.7–9 Cardiothoracic anesthesia may be an exception; faculty members from 30 randomly selected cardiothoracic anesthesia fellowship programs contributed 28 ± 46 publications on average, with senior faculty having more than junior faculty and ACGME-accredited fellowship program faculty having more than those in nonaccredited programs.2 Still, the 15 faculty directing non-ACGME cardiothoracic anesthesia fellowships had 20 ± 24 publications,2 which is several fold more than the directors of ACGME-accredited core anesthesia residency programs included in our study.

Moreover, in terms of scholarship, chairs of anesthesiology follow the same pattern compared with their counterparts in surgery, with the former having fewer than 40% as many PubMed citations and being less than half as likely to have had any type of NIH funding during their career as the latter.10 Indeed, in 2006 when the study was done, almost 1 in 5 academic anesthesiology chairs had 10 or fewer career publications.10 This is pertinent to the present work for a few reasons. First, inasmuch as anesthesiology faculty are generally academically weaker than surgical faculty, the pool of academically qualified faculty from which to choose a program director is likely to be smaller in anesthesiology than it is in surgery. Second, with more scholarly credentials, chairpersons of surgery may understand and value scholarship more and be in a better position than some chairs of anesthesiology to emphasize it and mentor their program directors in this regard once they assume the post.

Indeed, the low scholarly activity of anesthesiology residency program directors may simply mirror the weak academic environment of anesthesiology as a whole. This path might begin with anesthesiology residency itself. Many surgical residency programs require that trainees spend 1 or more years in research whereas few anesthesiology programs do. With no background in scholarship, junior faculty tend not to be productive in traditional academic endeavors even when enrolled in a formal departmental faculty development program.11 Whatever the reason, failure to establish a strong foundation in research among our trainees and junior faculty, or to recruit and retain those with such an interest, may relegate our specialty to perpetually underperform our more academically oriented colleagues in other specialties.

One implication of the discrepancy between the scholarship record of surgery and anesthesiology program directors is that it suggests the criteria and expectations for scholarship in program directors are not applied uniformly across these specialties by institutional offices and national organizations (e.g., Designated Institutional Official, local Graduate Medical Education Office) or the ACGME RRCs that are responsible for approving and sanctioning program director appointments. Of course, it could be that expectations for program director scholarship are sufficiently modest that anesthesiologists satisfy them and surgeons easily exceed them. Either way, greater transparency and guidance around expectations for program director scholarship at the ACGME and RRC levels are necessary. Along similar lines, the fact that program directors of anesthesiology had fewer publications in PubMed at each academic rank than surgery program directors (Fig. 1) suggests that institutions have dissimilar criteria, review processes, or mechanisms for academic promotion of physicians in different specialties. This practice appears to favor academic anesthesiologists, although we cannot exclude the possibility that surgeons and anesthesiologists get promoted in different academic series/tracks.

Figure 1
Figure 1:
Number of publications by academic rank among program directors of anesthesiology or surgery residency programs. Data are expressed as mean ± SD with 5% and 95% confidence intervals and analyzed using a 2-way analysis of variance (ANOVA) demonstrating a significant difference (P < 0.0001) in publications at each academic rank between the 2 departments.

One of the more interesting findings is that although a program director is first and foremost an educator, few of their peer-reviewed publications (5%–7%) reflect education-related work. In fact, approximately 75% and 40% of anesthesiology and surgery program directors, respectively, have no education-related publications. This irony is difficult to explain, although it may be related to lack of scientific training, unfamiliarity with use of psychometrics, a perception among program directors that research on educational programs is unnecessary or less professionally rewarding than other types of research, a tendency to publish in other forums such as newsletters or abstracts at educational meetings, or that peer-reviewed journals cited by PubMed consider work on educational innovation to have a lower priority for publication. Regardless, the impact of advances in education of anesthesiology and surgery residents will be limited unless they find their way into peer-reviewed journals. There is no shortage of worthy education-related topics to study. Most of the recent changes in postgraduate medical education are driven by educational theory rather than empirical evidence, and, as such, there is both ample opportunity and great need to test the validity and efficacy of such practices by the rigorous standards required for publication in peer-reviewed journals.

Our methodology has important weaknesses. As noted, the metrics of scholarly activity were NIH funding and peer-reviewed publications. Clearly, this misses other areas of legitimate academic achievement and scholarly work in which program directors might engage (e.g. contributions to books, syllabi, educational software, or web-based educational materials). Furthermore, because we defined education-related work narrowly, we may not have credited all educationally relevant publications (e.g., simulation, training, and mentoring) to this category. Similarly, NIH funding may not be a suitable metric for an educator; funding rates were low for program directors in both specialties, possibly because NIH funding for educationally related projects is limited. Furthermore, the NIH RePorter database lists only the principal investigator, so participation as a coinvestigator or expert consultant would not be captured. Similarly, we did not include other federal sources of grant funding (Department of Defense and Veteran’s Administration) or nonfederal sources of funding such as those offered by foundations or industry, though these are rarely available for innovative educational projects. It is unlikely, however, that any of these potentially confounding factors would disproportionately affect or bias the results in favor of one group relative to the other. In fact, a major strength of the study design is that the same criteria were applied to both groups; program directors were not only matched by institution but also held similar academic rank and had similar years in practice and tenure as a program director.

In summary, our study provides further evidence for a systemic weakness in the scholarly fabric of academic anesthesiology. On every measure of scholarly activity we examined, anesthesiology program directors fall short of their colleagues in surgery. What constitutes the minimum required level of scholarly activity among physicians to be an effective trainer of others is unknown, but this discrepancy between the specialties is worrisome because residency program directors are in a powerful position to mold the intellectual attitudes, aptitudes, and expectations of the next generation of physicians. If scholarly activity is essential for the advancement of medical science, improvement of postgraduate medical education, and enhancement of patient care within each subspecialty, then a program director must have credibility as a scholar. With medicine and medical education becoming ever more complicated and data more abundant, the need for scholarship in anesthesiology generally and anesthesia education specifically is greater than ever. The fact that a closely related specialty is better fulfilling this need makes it imperative that anesthesiology achieves the same high standard.

RECUSE NOTE

Dr. Gregory Crosby is the Section Editor for Neuroscience in Anesthesiology and Perioperative Medicine for the Journal. This manuscript was handled by Franklin Dexter, the Statistical Editor and Section Editor for Economics, Education, and Policy, and Dr. Crosby was not involved in any way with the editorial process or decision.

DISCLOSURES

Name: Deborah J. Culley, MD.

Contribution: This author helped design and conduct the study, analyze the data, and write the manuscript.

Attestation: Deborah J. Culley has seen the original study data, reviewed the analysis of the data, approved the final manuscript, and is the author responsible for archiving the study files.

Conflicts of Interest: Deborah J. Culley received honoraria from American Board of Anesthesiology. Dr. Culley is a Director of the American Board of Anesthesiology and a member of the ACGME Anesthesiology RRC.

Name: Brenda G. Fahy, MD.

Contribution: This author helped design and conduct the study and write the manuscript.

Attestation: Brenda G. Fahy has seen the original study data and approved the final manuscript.

Conflicts of Interest: Brenda G. Fahy received honoraria from American Board of Anesthesiology. Dr. Fahy is a Director of the American Board of Anesthesiology and a member of the ACGME Anesthesiology RRC.

Name: Zhongcong Xie, MD, PhD.

Contribution: This author helped write the manuscript.

Attestation: Zhongcong Xie has seen the original study data and approved the final manuscript.

Conflicts of Interest: The author has no conflicts of interest to declare.

Name: Robert Lekowski, MD.

Contribution: This author helped conduct the study and write the manuscript.

Attestation: Robert Lekowski has seen the original study data and approved the final manuscript.

Conflicts of Interest: The author has no conflicts of interest to declare.

Name: Sascha Buetler, MD, PhD.

Contribution: This author helped conduct the study and write the manuscript.

Attestation: Sascha Buetler has seen the original study data and approved the final manuscript.

Conflicts of Interest: The author has no conflicts of interest to declare.

Name: Xiaoxia Liu, MS.

Contribution: This author helped with statistical support and rewriting of the manuscript.

Attestation: Xiaoxia Liu has seen the original study data and approved the final manuscript.

Conflicts of Interest: The author has no conflicts of interest to declare.

Name: Neal H. Cohen, MD.

Contribution: This author helped design the study and write the manuscript.

Attestation: Neal H. Cohen has seen the original study data and approved the final manuscript.

Conflicts of Interest: Neal H. Cohen Former Chairman of the ACGME Anesthesiology RRC.

Name: Gregory Crosby, MD.

Contribution: This author helped design the study, analyze the data, and write the manuscript.

Attestation: Gregory Crosby has seen the original study data, reviewed the analysis of the data, and approved the final manuscript.

Conflicts of Interest: The author has no conflicts of interest to declare.

REFERENCES

1. Pagel PS, Hudetz JA. An analysis of scholarly productivity in United States academic anaesthesiologists by citation bibliometrics. Anaesthesia. 2011;66:873–8
2. Pagel PS, Hudetz JA. Scholarly productivity of United States Academic Cardiothoracic Anesthesiologists: influence of fellowship accreditation and transesophageal echocardiographic credentials on h-index and other citation bibliometrics. J Cardiothorac Vasc Anesth. 2011;25:761–5
3. Bould MD, Boet S, Sharma B, Shin E, Barrowman NJ, Grantcharov T. h-Indices in a university department of anaesthesia: an evaluation of their feasibility, reliability, and validity as an assessment of academic performance. Br J Anaesth. 2011;106:325–30
4. Hirsch JE. An index to quantify an individual’s scientific research output. Proc Natl Acad Sci U S A. 2005;102:16569
5. Zou G. A modified poisson regression approach to prospective studies with binary data. Am J Epidemiol. 2004;159:702–6
6. Schwinn DA, Balser JR. Anesthesiology physician scientists in academic medicine: a wake-up call. Anesthesiology. 2006;104:170–8
7. Benway BM, Kalidas P, Cabello JM, Bhayani SB. Does citation analysis reveal association between h-index and academic rank in urology? Urology. 2009;74:30–3
8. Choi M, Fuller CD, Thomas CR. Estimation of citation-based scholarly activity among radiation oncology faculty at domestic residency-training institutions: 1996–2007 Int J Radiat Oncol Biol Phys. 2009;74:172–8
9. Lee J, Kraus KL, Couldwell WT. Use of the h index in neurosurgery. Clinical article. J Neurosurg. 2009;111:387–92
10. Culley DJ, Crosby G, Xie Z, Vacanti CA, Kitz RJ, Zapol WM. Career National Institutes of Health funding and scholarship of chairpersons of academic departments of anesthesiology and surgery. Anesthesiology. 2007;106:836–42
11. Hindman BJ, Dexter F, Todd MM. Research, education, and nonclinical service productivity of new junior anesthesia faculty during a 2-year faculty development program. Anesth Analg. 2013;117:194–204
© 2014 International Anesthesia Research Society