Secondary Logo

What Every Graduating Resident Needs to Know About Quality Improvement and Patient Safety: A Content Analysis of 26 Sets of ACGME Milestones

Lane-Fall, Meghan, B., MD, MSHP; Davis, Joshua, J., MD; Clapp, Justin, T., PhD, MPH; Myers, Jennifer, S., MD; Riesenberg, Lee, Ann, PhD, RN, CMQ

doi: 10.1097/ACM.0000000000002039
Research Reports

Purpose Quality improvement (QI) and patient safety (PS) are broadly relevant to the practice of medicine, but specialty-specific milestones demonstrate variable expectations for trainee competency in QI/PS. The purpose of this study was to develop a unifying portrait of QI/PS expectations for graduating residents irrespective of specialty.

Method Milestones from 26 residency programs representing the 24 member boards of the American Board of Medical Specialties were downloaded from the Accreditation Council for Graduate Medical Education (ACGME) Web site in 2015. A codebook was generated by in-depth reading of all milestone sets by two authors. Using a content analytic approach, milestones were then coded by a single author, with a 25% sample double coded by another author. Descriptive statistics were used to characterize frequency counts.

Results Of 612 total milestones, 249 (40.7%) made mention of QI/PS. A median 10 milestones per specialty (interquartile range, 5.25–11.75) mentioned QI/PS. There were 446 individual references to QI, 423 references to PS, and another 1,065 references to QI/PS-related concepts, including patient-centered care, cost-effective practice, documentation, equity, handoffs and care transitions, and teamwork. QI/PS references reflected expectations about both individual-level practice (531/869; 61.1%) and practice within a health care system (338/869; 38.9%). QI and PS references were linked to all six ACGME core competencies.

Conclusions Although there is variability in the emphasis placed on QI/PS across specialties, overall, QI/PS is reflected in more than 40% of residency milestones. Graduating residents in all specialties are expected to demonstrate competence in QI, PS, and multiple related concepts.

M.B. Lane-Fall is assistant professor of anesthesiology and critical care and codirector, Center for Perioperative Outcomes Research and Transformation, Perelman School of Medicine, University of Pennsylvania, senior fellow, Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, and fellow, Harold Amos Medical Faculty Development Program, Robert Wood Johnson Foundation; ORCID:

J.J. Davis is emergency medicine resident, Penn State Hershey Medical Center, Hershey, Pennsylvania, and visiting scholar, American Board of Medical Specialties, Chicago, Illinois.

J.T. Clapp is research associate, Department of Anesthesiology and Critical Care, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania.

J.S. Myers is associate professor of clinical medicine, director, Center for Healthcare Improvement and Patient Safety, Perelman School of Medicine, University of Pennsylvania, and director of quality and safety education, Department of Medicine, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pennsylvania.

L.A. Riesenberg is professor of anesthesiology and perioperative medicine, University of Alabama at Birmingham, Birmingham, Alabama.

Funding/Support: None reported.

Other disclosures: None reported.

Ethical approval: Ethical approval for the reported work was not sought, as all data were previously published and publicly available.

Previous presentations: The work reported herein was previously presented in abstract form at the American Board of Medical Specialties annual conference, Cleveland, Ohio, September 28, 2016; and the American College of Medical Quality MQ2017 conference, Washington, DC, March 31, 2017.

Supplemental digital content for this article is available at

Correspondence should be addressed to Meghan Lane-Fall, 3400 Spruce St., 680 Dulles, Philadelphia, PA 19104; telephone: (215) 573-7399; e-mail:; Twitter: @mlanefall.

While “first do no harm” has always been a central tenet of the medical profession, the focus on patient safety (PS) and quality improvement (QI) has intensified following the release of To Err Is Human 1 and Crossing the Quality Chasm 2 from the Institute of Medicine. Indeed, a focus on PS and QI has become part of the fabric of successful health care systems. In recognition of this new reality, the Accreditation Council for Graduate Medical Education (ACGME) created two core competencies, practice-based learning and improvement (PBLI) and systems-based practice (SBP), which broadly include QI and PS content. The primary goal of the ACGME’s Next Accreditation System3 was to shift medical education from a content focus to an outcome focus. As a core part of this initiative, competency-based behavioral descriptions known as milestones3 further expanded on all six of the core competencies, including PBLI and SBP, in a specialty-specific manner.

Education and training in QI and PS should occur throughout the educational continuum.4 Even though the expected knowledge, skills, and attitudes within the domains of quality and safety are shared among physician trainees from different specialties, there is significant variability in the QI and PS milestones among the major specialties and subspecialties. While there is expected and necessary variability in the clinical expression of many of these competencies, there is presumably a core set of competencies in QI and PS in which all physicians, regardless of specialty, should be trained. For example, while engaging in practices designed to promote PS includes actively participating in time-outs prior to operative procedures, participating in operating room time-outs is more relevant to surgery and anesthesia than to nonsurgical specialties. In the interest of advancing the discussion about the necessary knowledge, skills, and attitudes in QI and PS for all graduating residents, we set out to identify common themes across milestones from 26 specialties recognized by the 24 member boards of the American Board of Medical Specialties (ABMS). The goal of this work was to develop a unifying portrait of QI and PS training goals that could inform a common set of milestones, the cultivation of entrustable professional activities5 (EPAs) reflecting progressive responsibility in performing quality and safety work in clinical medicine, and perhaps a common quality and safety curriculum for graduate medical education (GME).

Back to Top | Article Outline


This study is a content analysis of published milestones from the ACGME. Milestones are competency-based developmental outcomes for assessing graduate medical trainees. They contain specialty-specific subcompetencies that map to the six core competencies. Within each subcompetency is a description (anchor) of ways a trainee can demonstrate their performance, which can be mapped to five levels. Although the levels all have different meanings across specialties, in most specialties, Level 1 is where interns enter residency, Level 4 is a graduation target for residents, and Level 5 is an aspirational target.

Because the ACGME milestones are publicly available, we did not seek institutional review board approval for this study. In October 2015, and again in December 2015, we downloaded milestones representing specialties from the 24 ABMS member boards from the ACGME Web site.6 We found that 2 member boards each had 2 relevant major ACGME specialties: neurology and psychiatry under the American Board of Psychiatry and Neurology, and diagnostic radiology and radiation oncology under the American Board of Radiology. This resulted in 26 sets of milestones included in this study. We rechecked the ACGME Web site in May 2016 to look for recently published or updated milestones from these specialties, and we found no new milestones published since our original search.

To develop a framework for understanding the breadth of QI and PS expectations across all specialties, we performed a content analysis.7 We employed an integrative approach to coding, incorporating both inductive and deductive elements.8 For the inductive approach, two authors (J.J.D. and M.L.F.) read through all 26 sets of milestones to identify individual milestones relating to QI and PS. This initial review enabled us to develop and define categories relating to QI and PS in clinical training. For the deductive approach, we used the Institute of Medicine (now National Academy of Medicine) six quality aims2 to develop our definitions of QI and PS. We defined “quality improvement” as a formal approach, using QI methods and tools to analyze performance and systematically improve the quality of patient care in any of the six Institute of Medicine domains: safety, timeliness, effectiveness, equity, efficiency, and patient-centeredness. Analyses of individual or team-based clinical quality data and improvement efforts using plan–do–study–act approaches are examples of activities included in the QI category. We defined “patient safety” as activities meant to decrease health-care-related harm to patients. Error prevention activities as well as endeavors meant to explain or deconstruct actual or potential PS lapses, such as root cause analyses, were included in the PS category.

Using the initial inductively derived codebook, we coded by consensus, to categorize all the milestones and to ensure coding consistency. During later rounds of coding, we subdivided milestone narrative comments classified as QI or PS into those focused on individual provider behavior (e.g., “Role models continuous quality improvement of personal practice … using advanced methodologies and skill sets”—family medicine) and those focused on system behavior or functioning (e.g., “Designs measurement tools to monitor and provide feedback to providers/teams on resource consumption to facilitate improvement”—thoracic surgery).

In addition to coding QI and PS, we identified “related concepts” consistent with the Institute of Medicine aims, but conceptually distinct from QI and PS. These included cost-effectiveness, documentation, equity (including ethical and culturally effective care), handoffs and transitions in care, patient-centered care, and teamwork. The final codebook included 10 categories: QI—individual; QI—systems; PS—individual; PS—systems; cost-effectiveness; documentation; equity; handoffs and transitions in care; patient-centered care; and teamwork.

For this project, a “reference” was any direct mention of a behavior, knowledge area, skill, or attitude directly related to a QI or PS activity, and could occur at zero, one, or multiple levels within a given milestone. For example, if a QI and PS concept was mentioned in all levels of a given milestone, it would be counted as five references. Each level of individual milestones could be coded to multiple categories. All coding that preceded establishment of the final codebook was done by consensus, obviating calculation of interrater reliability. Once the final codebook was established, one author (J.J.D.) assigned final codes independently. A random sample of 25% of these codes was checked by a different author (M.L.F.).

Project data were managed using Microsoft Excel 2016 (Microsoft Corporation, Redmond, Washington). Descriptive statistical analyses were conducted with Stata statistical software, version 13 (StataCorp LLC, College Station, Texas).

Back to Top | Article Outline


Descriptive analysis of all milestones

In the 26 specialties, there were 612 total milestones (median, 22.5; interquartile range [IQR], 20.25–26.75; range, 10–41), each with five levels as required by the ACGME. There was also an appendix of 9 “nonreported” milestones in 1 specialty (physical medicine and rehabilitation). There was appreciable heterogeneity in milestone formatting, with use of prose, short phrases, or bulleted lists to characterize both milestone descriptions and level descriptions. A few specialties offered examples of behaviors in each level assignment (e.g., pediatrics, psychiatry, urology). In general, there were more milestones designated as patient care (PC) than as the other competencies (Table 1). Medical knowledge (MK) and PC milestones were more detailed than those addressing the other four competencies.

Table 1

Table 1

For some specialties, the milestones for PBLI, SBP, professionalism (Prof), and interpersonal and communication skills (ICS) were listed with one overarching heading with gradations of achievement. For example, a single narrative description for the milestone was given, with Level 1 assigned if the trainee exhibited the target behaviors “< 25% of the time” or “rarely” and Level 4 assigned if the trainee exhibited the target behaviors “> 80% of the time” or “consistently” (e.g., allergy and immunology; colon and rectal surgery).

Representation of QI- and PS-related milestones also varied. Within the 26 sets of milestones (612 individual milestones), 249 (40.7%) included mentions of QI or PS (interrater agreement on coding of a 25% random sample of milestone references was 95.4% [474/497]). There were a median 10 milestones per specialty with any reference to QI and PS (IQR, 5.25–11.75), or 41% of each specialty’s milestones (IQR, 29%–51%). Overall, there was a total of 1,934 references to quality and safety concepts, of which 869 were directly related to quality and PS (Table 1), and 1,065 concerned QI/PS-related concepts (Table 2). Many of these references were located in milestones assigned to the core competencies of PBLI or SBP, but they also appeared in the other core competencies: MK, PC, Prof, and ICS (Table 2).

Table 2

Table 2

In-depth reading enabled characterization of QI/PS milestones as reflecting knowledge, skills, or attitudes (Supplemental Digital Appendices 1 and 2 at We also noted that, within each of these domains, there were two basic focal points of QI and PS milestones—an individual focus and a systems focus. That is, some milestones described knowledge, skills, and attitudes about individual physician practice (“Incorporates QI into clinical practice”—diagnostic radiology), whereas others were focused on health care at a systems level (“Viewed as a leader in identifying and advocating for the prevention of medical error”—internal medicine) (Table 3). There were more QI and PS references focused on individual improvement (531/869; 61%) than systems improvement (338/869; 39%).

Table 3

Table 3

Back to Top | Article Outline

Content of milestones reflecting QI, PS, and related concepts

Quality improvement.

Of the 612 milestones, 140 (26.1%) made mention of QI concepts. Within these 140 milestones, we counted 446 QI references (243 individual focused and 203 systems focused). Comparing counts for QI references across levels, more QI references were clustered at Level 5 (beyond graduation target) than would be expected with even distribution across levels (61/243 [25.1%] for individual, 67/203 [33.0%] for systems level; Figure 1).

Figure 1

Figure 1

Subthemes for individual-focused QI were often found in the PBLI milestones (Table 1). These subthemes included incorporating national guidelines into practice, identifying deficits in one’s own practice, participating in benchmarking, and a commitment to lifelong improvement. For systems-focused QI, common subthemes included knowledge of QI terminology and processes, identifying system flaws or workarounds, engaging in continuous improvement, and leading a QI team.

Back to Top | Article Outline

Patient safety.

Of the 612 milestones, 134 (25%) made mention of PS concepts. Within these 134 milestones, we counted 423 references, 288 (68.1%) individual focused and 135 (31.9%) systems focused. Comparing counts for PS references across levels (Figure 1), almost half of the individual-focused references were located at Levels 1 and 2 (142/288; 49.4%). There was no discernible pattern for systems-focused milestones in PS.

Common subthemes for individual-focused PS included engaging in infection prevention procedures, reporting errors, and identifying and preventing errors in one’s own practice. Common subthemes in systems-focused references included defining PS terminology, participating in error analysis exercises, contributing to morbidity and mortality conferences, and leading an interprofessional team to investigate PS issues.

Back to Top | Article Outline

Related concepts.

We also identified six concepts that were related to high-quality, safe patient care: teamwork, cost-effectiveness, patient-centered care, equity (including culturally sensitive practice), documentation, and handoffs and transitions in care. The number of references of each of these concepts is as follows: teamwork, 329; cost-effectiveness, 171; patient-centeredness, 162; equity, 161; documentation, 133; and handoffs, 109. We identified exemplar milestone statements for each of these related concepts as well as the previously identified four major domains relating to QI and PS (Table 3).

Themes from the QI/PS domains and related concepts form a core set of competencies that are relevant to all trainees in GME irrespective of specialty (Table 4).

Table 4

Table 4

Back to Top | Article Outline


Despite significant variability in the description of ACGME milestones relating to QI and PS, each of the 26 specialties includes a substantial number of milestones that contain these topics broadly relevant to physicians in training. Specifically, graduating residents are expected to understand how QI, PS, and related concepts should be embedded into their clinical practice and how they affect health system functioning. Although historically QI/PS content was thought to be confined within the competencies of SBP and PBLI, our content analysis found that 33.7% (293/869) of the references to knowledge, skills, and attitudes in these topic areas were in the other four core competencies: 152 in PC, 63 in Prof, 51 in MK, and 27 in ICS. This underscores the important point that teaching and learning about QI and PS should not be thought of as something separate from patient care and the other competencies but, rather, integrated into other experiences whenever possible. Such integration will help trainees internalize the requisite knowledge, skills, and attitudes to practice safely and deliver high-quality health care within a complex system.

Our findings highlight that QI and PS bridge all medical specialties. As such, substantive training in QI and PS should start in medical school. The Liaison Committee for Medical Education Medical School Objectives Project emphasizes several important aspects of quality care and endorses universal training on these topics throughout the medical education curriculum, while the Association of American Medical Colleges (AAMC) has developed core EPAs for graduating medical students entering residency that include QI/PS skills.9 Rigorous medical school training in QI/PS would provide entering residents with a stronger foundation on which to build competency in QI/PS as they master their individual specialties.

Curricular interventions have shown improvements in QI and PS knowledge, skills, and attitudes.9 , 10 And although the methods are variable and the research quality is limited, research on some interventions has shown systemic process changes in care delivery at the local level.9 , 10 However, before comprehensive curricula can be designed, agreement on educational goals is needed. Indeed, the lack of a uniform set of competencies in QI/PS has created frustration for some medical educators because it can cause confusion about what to teach. The variability in QI/PS competencies across specialties that we found affirms this frustration. Development of a common set of competencies, perhaps with a unified curriculum, would not only streamline resident education but also facilitate the goal of training undergraduate medical trainees in QI/PS.

It is interesting to note that, within the milestones, a significant proportion of the QI references were clustered at Level 5—beyond the currently designated graduation target for most residents—while a significant portion of the individual-focused PS references were clustered at Level 1 (71/288; 24.7%) and Level 2 (71/288; 24.7%). We believe that this reflects the number of PS competencies that are foundational for an individual to practice safely in the clinical learning environment and are thus required to be fully present or at least developing at the start of internship. Such foundational skills and attitudes include speaking up for safety concerns, collaborating as a member of an interprofessional team, identifying system failures, and transitioning patient care in a responsible manner. These competencies should be taught in medical school, and indeed many of them are included in the AAMC’s core EPAs for entering residency.9

We hypothesize that fewer of the QI competencies are focused at these early levels of GME training for two reasons. First, to reflect on one’s clinical practice with a goal of improvement, one must have acquired a certain amount of clinical experience over time. Similarly, practicing within a health care system for a certain period of time is necessary in order to understand and improve on it, and many interns are entering a system that is new to them as they start their residency. Second, when compared with the PS competencies, the activities necessary to attain competence in systems-focused QI require a new set of knowledge and skills related to QI tools and measurement. The clinical demands of internship may preclude many specialties from integrating these activities into their intern curriculum. Although educating trainees in QI in the mid to later years of GME training may be appropriate, it is interesting to note that a quarter (25%) to a third (33%) of the individual- and systems-focused QI competencies were listed at Level 5, which is considered aspirational and beyond the graduation-level target. This could represent the fact that more advanced competencies in QI are being recognized and incorporated into the growing number of advanced training opportunities or concentrations in QI offered around the country,10 , 11 but it is problematic if QI competency is expected at residency graduation. We believe that all residents should demonstrate some level of competency in QI by graduation, and we suggest that some of the QI milestones currently designated as Level 5 be shifted to lower competency levels.

Our project has important limitations. First, we analyzed currently published milestones representing the 24 ABMS member boards. Given that the milestone sets continue to evolve, our analysis may not have captured work in progress relevant to QI and PS. Second, we included only core specialties, so our analysis did not capture ACGME-approved fellowships, some of which have created their own milestones. Third, we were not able to analyze the implementation of milestones in GME programs. How these milestones are applied in practice will inevitably differ from how they are described and organized in the ACGME residency review committees’ milestone sets. Finally, the content analysis we pursued here is reductive: As our central intent was synthetic—to measure the frequency and location of QI/PS milestones across specialties—we sacrificed some qualitative nuance in unpacking the diverse ways in which QI- and PS-related concepts were expressed by the various ABMS authors.

Despite these limitations, we believe that this work adds value to the field of medical education by elucidating how QI, PS, and related competencies are organized across milestones and thus conceptualized across specialties. The differences we found between specialties suggest that QI and PS are not viewed uniformly. These differences hinder our ability to train and reinforce consistent QI and PS learning for medical students and residents. By highlighting similarities between specialties, we hope to advance the discussion about development of common milestones and, ultimately, common curricula that could be shared or deployed across specialties. This is an important goal, as QI/PS is still relatively new to medical educators, and faculty expertise is lacking.4 , 12 , 13 By emphasizing similarities and highlighting differences, our intent is to move the field toward a consensus on what every graduating resident needs to know in QI and PS.

Back to Top | Article Outline


Graduating residents from 26 ACGME-accredited specialties are expected to be competent in QI and PS and incorporate knowledge, skills, and attitudes from these fields into their clinical practice. While significant variability is present in the QI and PS expectations among the ACGME specialty milestones, there are many common themes that permeate throughout and support the need for a common set of educational outcomes for QI and PS in GME.

Acknowledgments: The authors would like to acknowledge Frances K. Barg, PhD, MEd, who assisted with study design.

Back to Top | Article Outline


1. Institute of Medicine. To Err Is Human: Building a Safer Health System. 2000.Washington, DC: National Academies Press.
2. Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. 2001.Washington, DC: National Academies Press.
3. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—Rationale and benefits. N Engl J Med. 2012;366:1051–1056.
4. Association of American Medical Colleges. Teaching for quality: Integrating quality improvement and patient safety across the continuum of medical education. Report of an expert panel. Published 2013. Accessed October 24, 2017.
5. Ten Cate O. Nuts and bolts of entrustable professional activities. J Grad Med Educ. 2013;5:157–158.
6. Accreditation Council for Graduate Medical Education. Milestones. Accessed September 26, 2017.
7. Hays DG, Singh AA. The Basics of Qualitative Data Management and Analysis: Qualitative Inquiry in Clinical and Educational Settings. 2011.New York, NY: Guilford Press.
8. Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: Developing taxonomy, themes, and theory. Health Serv Res. 2007;42:1758–1772.
9. Association of American Medical Colleges. Core entrustable professional activities for entering residency: Curriculum developers’ guide. Published 2014. Accessed October 24, 2017.
10. Patel N, Brennan PJ, Metlay J, Bellini L, Shannon RP, Myers JS. Building the pipeline: The creation of a residency training pathway for future physician leaders in health care quality. Acad Med. 2015;90:185–190.
11. Karasick AS, Nash DB. Training in quality and safety: The current landscape. Am J Med Qual. 2015;30:526–538.
12. Baxley EG, Lawson L, Garrison HG, et al. The Teachers of Quality Academy: A learning community approach to preparing faculty to teach health systems science. Acad Med. 2016;91:1655–1660.
13. Myers JS, Tess A, Glasheen JJ, et al. The Quality and Safety Educators Academy: Fulfilling an unmet need for faculty development. Am J Med Qual. 2014;29:5–12.

Supplemental Digital Content

Back to Top | Article Outline
© 2018 by the Association of American Medical Colleges