The Evolving Curriculum in Quality Improvement and Patient Safety in Undergraduate and Graduate Medical Education: A Scoping Review : American Journal of Medical Quality

Secondary Logo

Journal Logo

Review Article

The Evolving Curriculum in Quality Improvement and Patient Safety in Undergraduate and Graduate Medical Education: A Scoping Review

Li, Chris J. BS1; Nash, David B. MD, MBA2

Author Information
doi: 10.1097/JMQ.0000000000000084
  • Free

Abstract

Introduction

The problem of health care quality was brought to national attention in the early 2000s through the publication of To Err is Human and Crossing the Quality Chasm, which demonstrated the need for a comprehensive approach to improve patient safety and to close the quality gap in the United States health care system.1,2 Both the Accreditation Council for Graduate Medical Education (ACGME) and Association of American Medical Colleges (AAMC) recognized this priority, and subsequently increased the emphasis on physician development in the Quality Improvement and Patient Safety (QIPS) domains.3

While several QIPS curriculums have been developed in the past two decades, in 2018 the Clinical Learning Environmental Review (CLER) program reported that QIPS programs in GME were unsuccessful in the transmission of QI knowledge and practical experiences to residents.4 Despite increased attention on QIPS, average quality performance indicators have not improved and this cannot be attributed solely to the COVID-19 pandemic.5 Thus, an even greater focus on the development of a strong standardized curriculum is required. There have been several curriculum guides previously published that guide both medical schools and residency programs through the development process.6 However, the variability among curriculums, poor performance, and the CLER 2.0’s push for improvement of QIPS programs highlights the need for an updated evaluation and analysis of undergraduate medical education (UME) and graduate medical education (GME) curriculum development and performance.4

Given the heterogenous nature of the QIPS curriculums in the United States, a scoping review was performed to evaluate the current state of non-degree QIPS programs in both UME and GME. The review does not analyze accredited QIPS graduate programs given their priority and focus on developing leaders in the field of health care quality.7 Instead, this review provides both an update on the recent literature findings regarding curriculum development and highlights specific curriculum components that characterize successful non-degree programs, whose focus mainly lies within improving QIPS skill and competency.

Methods

The PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) model was used as a guide to develop this scoping review. This model is an extension of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) framework. The PRISMA-ScR is an evidence-based framework developed as the quality standard for the construction of scoping reviews.8 A database search of PubMed and Scopus was conducted in March 2022. The search queries for both databases were developed utilizing keywords from 7 relevant articles provided by field expert (D.N.). A librarian with experience in search query development was used to optimize the search criteria and the following queries were finalized:

  1. PubMed: ((((“Learning Health System”[Mesh]) OR (“Fellowships and Scholarships”[Mesh])) OR (“Schools, Medical”[Mesh])) OR (“Education, Medical, Graduate”[Mesh])) AND ((“Quality Improvement”[Mesh]) OR (“Patient Safety”[Mesh])) Filters: English, from 2019 to 2022.
  2. SCOPUS: (TITLE-ABS-KEY (“medical education”) OR TITLE-ABS-KEY (“fellowships and scholarships”) OR TITLE-ABS-KEY (“internship and residency”) AND ABS (“patient safety”) OR TITLE-ABS-KEY (“quality improvement”)) AND (LIMIT-TO (AFFILCOUNTRY, “United States”)) AND (LIMIT-TO(PUBYEAR, 2022) OR LIMIT-TO (PUBYEAR, 2021) OR LIMIT-TO (PUBYEAR, 2020) OR LIMIT-TO (PUBYEAR, 2019)) AND (LIMIT-TO (DOCTYPE, “ar”) OR LIMIT-TO (DOCTYPE, “re”)).

Eligibility criteria included articles restricted to articles published between January 2019 and March 2022, studies exclusive to the US UME and GME programs, and physician-only curriculums. The 2018 CLER 2.0 report that QIPS curriculum were largely ineffective guided the authors’ decision to exclude articles published before the year 2019 to allow for evaluation of new data and information not previously studied from a comprehensive lens. Non-US studies were also excluded given the unique complexity of the US health care system and decreased applicability of international curriculums to the US UME and GME structures. Finally, the study was restricted to physician-only programs as interdisciplinary programs decrease the applicability and skew the results of curriculum analyses specific to physician QIPS education.

The search results were screened by primary reviewer (C.L.) for related titles and abstracts, and the full-text of the remaining articles were screened for eligibility first by primary reviewer (C.L) and again, by secondary reviewer (D.N). The remaining articles were summarized and characterized by program characteristics and targeted population.

Results

The multiple-database search generated 1,252 articles (SCOPUS = 738 and PubMed = 514). Duplicates (n = 177) were identified and removed using the EndNote 20.3 software which yielded 1,075 unique articles. The remaining articles were screened by title and abstract yielding 109 articles identified as relevant to QIPS education in UME and GME. The full-texts of the remaining 109 articles were assessed for applicability and to ensure all inclusion criteria were met. Sixty-three of the 109 articles were excluded, the most common reasons for exclusion were unrelated topics (n = 43), international studies (n=9), and interprofessional curriculums (n = 9). The reviewers performed a citation search from the remaining 46 eligible articles. One relevant study was found from the citation search, which ultimately resulted in 47 articles deemed appropriate for inclusion (see Figure 1).

F1
Figure 1.:
PRISMA-ScR search framework. Illustration of the search and study selection process for a scoping review of the literature on the evaluation of undergraduate and graduate quality improvement and/or patient safety curriculum programs published between January 2019 and March 2022.

A description and evaluation of the included articles contents are demonstrated in the Table. Of the 47 articles that met inclusion criteria, 87.2% (n = 41) were Original Research Articles, 8.5% (n = 4) were Review articles and the remaining 2 articles were categorized as “Other” (reflections, commentaries). Eighty-one percent (80.9%, n = 38) focused on curriculum development at the GME level while the other 19.1% (n = 9) targeted UME curriculum development. The articles were further characterized a Specialty Specific, meaning curriculums were developed for specific clerkship rotations or a particular residency/fellowship cohort. Seventy percent (70.2%, n = 33) of the included articles were specialty specific with the most common clerkships and specialties of focus being Internal Medicine (n = 8) and Surgery (n = 6). Figure 2 demonstrates the distribution of specialties in the included articles that were characterized as “Specialty”-specific.

Table. - All Eligible Articles and Related Content.
Title PS/QI/Both Structure Evaluation Outcome Year UME/GME Type of study General/specialty Specialty
Patient safety education in anesthesia: Current state and future directions 9 PS x 2021 GME Review Specialty Anesthesiology
Resident and fellow engagement in safety and quality 10 Both x 2019 GME Review Specialty ENT
Evaluation of a quality improvement experience for family medicine clerkship students 11 QI x x x 2021 UME Original Research Specialty Family Medicine
A longitudinal curriculum for quality improvement, leadership experience, and scholarship in a family medicine residency program 12 QI x x x 2020 GME Original Research Specialty Family Medicine
Students adding value: improving patient care measures while learning valuable population health skills 13 QI x x x 2020 UME Original Research Specialty Family Medicine
Use of a rubric to improve the quality of internal medicine resident event reporting 14 PS x x x 2021 GME Original Research Specialty Internal Medicine
Comparing the effects of design thinking and a3 problem-solving on resident attitudes toward systems change 15 QI x x x 2021 GME Original Research Specialty Internal Medicine
Evolution of a resident quality improvement curriculum: lessons learned on the path from innovation through stability to contraction 16 Both x x 2019 GME Original Research Specialty Internal Medicine
Swimming with sharks: teaching residents value-based medicine and quality improvement through resident-pitched projects 17 QI x x x 2020 GME Original Research Specialty Internal Medicine
“I think I was losing the forest for the trees”: evaluation of an internal medicine residency quality improvement curriculum 18 QI x x x 2020 GME Original Research Specialty Internal Medicine
QIPS CURE: implementing a quality improvement and patient safety curriculum and resident experience 19 Both x x x 2020 GME Original Research Specialty Internal Medicine
A longitudinal ambulatory quality improvement curriculum that aligns resident education with patient outcomes: a 3-year experience 20 QI x x x 2020 GME Original Research Specialty Internal Medicine
Re-thinking how we teach quality improvement: adding meaning to an ACGME requirement 21 Both x x x 2019 GME Original Research Specialty Internal Medicine
A quality improvement curriculum for the neurology clerkship: A practice-based approach to discharge education 22 QI X X X 2019 UME Original Research Specialty Neurology
Education Research: a novel resident-driven neurology quality improvement curriculum 23 QI x x x 2020 GME Original Research Specialty Neurology
To the point: Integrating patient safety education into the obstetrics and gynecology undergraduate curriculum 24 PS x x 2020 UME Review Specialty OB/GYN
QI Bootcamp: feasibility and acceptability of a novel approach to training residents in process improvement 25 QI x x x 2019 GME Original Research Specialty OB/GYN
ASCO quality training program: a five-year review 26 QI x x x 2020 GME Original Research Specialty Oncology
Implementation of a quality and patient safety curriculum for pathology residency training 27 Both x x x 2021 GME Original Research Specialty Pathology
Development of a quality improvement curriculum in a pediatric emergency medicine fellowship 28 QI x 2019 GME Original Research Specialty Pediatric Emergency Medicine
A national patient safety curriculum in pediatric emergency medicine 29 PS x x x 2019 GME Original Research Specialty Pediatric Emergency Medicine
Pediatric quality improvement (QI) virtual practicum: adapting a QI simulator 30 QI x x x 2020 GME Original Research Specialty Pediatrics
Physical medicine and rehabilitation residency quality improvement and research curriculum: design and implementation 31 QI x x x 2021 GME Original Research Specialty PM&R
A quality improvement curriculum for psychiatry residents 32 QI x x x 2020 GME Original Research Specialty Psychiatry
A novel experiential quality improvement training program during residency improves quality improvement confidence and knowledge: a prospective cohort study 33 QI x x x 2020 GME Original Research Specialty Psychiatry
Implementing a radiology residency quality curriculum to develop physician leaders and increase value for patients 34 Both x x 2020 GME Original Research Specialty Radiology
A longitudinal course pilot to improve surgical resident acquisition of quality improvement skills 35 QI x x x 2021 GME Original Research Specialty Surgery
Evaluation of changes in quality improvement knowledge following a formal educational curriculum within a statewide learning collaborative 36 QI x x x 2020 GME Original Research Specialty Surgery
QIC: a pilot quality improvement curriculum utilizing interactive, team-based workshops 37 QI x x x 2021 GME Original Research Specialty Surgery
Defining the content for a quality and safety in surgery curriculum using a nominal group technique 38 Both x 2019 GME Original Research Specialty Surgery
Aggregated student confidence estimates support continuous quality improvements in a competencies-oriented curriculum 39 QI x x 2019 GME Original Research Specialty Surgery
A structured quality improvement educational curriculum increases surgical resident involvement in QI processes 40 QI x x x 2020 GME Original Research Specialty Surgery
Important elements in the quality improvement curriculum for orthopaedic residents 41 QI x x x 2019 GME Original Research Specialty Surgery (Orthopaedic)
Career impact of the chief resident in quality and safety training program: an alumni evaluation 42 Both x x x 2020 GME Original Research General
A guide to evaluation of quality improvement and patient safety educational programs: lessons from the va chief resident in quality and safety program 43 Both x x 2019 GME Original Research General
What we learned in the development of a third-year medical student curricular project 44 QI x 2021 UME Other General
Outcomes of a longitudinal quality improvement and patient safety preclerkship curriculum 45 Both x x x 2019 UME Original Research General
Teaching students how to improve safety and quality in two children’s hospitals: building a pediatric clerkship patient safety and quality experience 46 PS x x 2019 UME Original Research General
Teaching students to transform: the health innovations scholars program 6 years later 47 Both x x x 2020 UME Original Research General
Impacting the next generation: teaching quality and patient safety 48 Both x x 2019 GME Review General
Feasibility and effectiveness of a quality improvement curriculum for combined medicine subspecialty fellows 49 QI x x x 2022 GME Original Research General
Simulation use in quality improvement and patient safety training for residents 50 Both x x 2021 GME Original Research General
A quality education: a comprehensive review of a combined longitudinal and specialty track quality improvement and patient safety medical school curriculum 51 Both x x 2020 UME Original Research General
Demonstrating the value of postgraduate fellowships for physicians in quality improvement and patient safety 52 Both x x x 2020 GME Original Research General
Expanding training in quality improvement and patient safety through a multispecialty graduate medical education curriculum designed for fellows 53 Both x x x 2020 GME Original Research General
Quality improvement education: redesigning intermountain healthcare’s advanced training program for a value-based learning health care system 54 QI x 2021 GME Other General
Integrating physicians into lean quality improvement through a structured educational program the ECLIPSE program 55 QI x x x 2022 GME Original Research General
Abbreviations: QI, quality improvement; PS, patient safety.

F2
Figure 2.:
Graph of specialty distribution of all specialty-specific QIPS programs. The graph depicts the distribution of the 33 different specialty-specific QIPS programs. UME QIPS programs that were implemented during a particular clerkship were categorized as “specialty-specific.” Abbreviation: QIPS, Quality Improvement and Patient Safety.

Regarding the content and focus of QIPS education, 55.3% (n = 26) of all articles focused exclusively on Quality Improvement (QI), 10.6% (n = 5) focused exclusively on Patient Safety (PS), and the remaining 34.1% (n = 16) focused on both QI and PS. The overwhelming content of most articles included an overview of the curriculum structure, methods of evaluation, and outcomes of the respective program. More specifically, nearly all articles (n = 46) cover curriculum structure, while the majority also cover evaluation metrics (n = 36) and or program outcomes (n = 36). This is demonstrated below in Figure 3. While some articles also highlighted specific competencies of focus in their respective curriculums, the heterogeneity of these competencies and smaller body of evidence guided the authors’ decision to not include this topic in the discussion.

F3
Figure 3.:
Distribution graph of studies based on curriculum content. The 3 categories were not mutually exclusive to each other, each article was eligible to be included in each of the 3 categories.

Discussion

Structure

Takeaways from the Nonspecific General Programs: Combination and Crossover Are Key

In terms of structure, nondegree programs were a mix of specialty-specific and general, as demonstrated in Figure 4. General programs at the GME level ranged significantly in their target demographic and as a result, their curriculum structure.

F4
Figure 4.:
Focus of each curriculum. The focus of each curriculum, grouped by either “General” curriculums which represented multispecialty programs or UME programs that spanned multiple clerkships termed “Specialty-specific,” which targeted a specific specialty or UME clerkship.

Program length ranged more significantly in GME-General programs. The shortest 2 programs were limited 2 simulation-based QIPS sessions50 and 4 online virtual QIPS modules.55 A few nondegree GME programs were more formal, described as single or multiyear time-intensive curriculum that occupied the majority of residents (or fellows) schedules, including a 1-year chief-resident quality and safety programs42,43 and a 2-year dedicated QIPS fellowship.52 While these programs may be difficult to implement on a widespread national level for all training-physicians given the time commitment and resources required, it is an excellent avenue to facilitate the development of leadership and faculty expertise in QI. Longitudinal curriculums that were integrated into residency program and not dedicated to a particular specialty demonstrated two important takeaways. First was the feasibility of extending a QI curriculum to multiple divisions across a hospital, effectively allowing for a consolidation of scarce QIPS resources.49 The “general” or multispecialty inclusive curriculums also successfully addresses the insufficient faculty experience with QIPS teaching. Centralizing QIPS training is a novel strategy to effectively utilize the value of faculty with QIPS experience.46,47 The UME-General programs were slightly more uniform than their GME counterparts, most exhibiting a longitudinal curriculum that spanned the preclerkship years with more targeted experiential activates initiated in their clerkship curriculum.45,46,49 While this may be in part due to the uniform nature of UME and difficulty in establishing “longitudinal” specialty-specific curriculums during 3- to 8-week clerkships, these programs demonstrated the benefits of a longitudinal, multimodal approach allowing for spaced repetition and reinforcement necessary to maximize long-term retention.43

Takeaways from Specialty Specific Curriculums: Uncovering Common Components: (Experiential) Learning, Length, and Lectures

The wealth of literature focused on specialty-specific UME and GME programs highlights similar curriculum components seen across multiple specialties that have contributed to successful QIPS programs. The correlation of these components to successful programs suggest their value and utility in QIPS program development.

First, a common theme focused on the value of experiential learning. More specifically, several articles assert that experiential learning was a valuable complement to didactic education and integral to the success of QIPS curriculums.10,15,21,36 Expanding on experiential learning, engagement from physician proponents and faculty support seems to be integral in facilitating QIPS improvement and QI project completion.34 Without faculty and curriculum support, QI project completion was noted to dimmish.16 In addition to faculty oversight, resident involvement was integral in utilization of experiential learning. Specifically, resident engagement in design, implementation, analysis, and presentation of projects was a crucial component in gaining QI skill through these experiential learning projects.17,33 Participants also highlighted their preferences for team-based projects.19,20,37,51 The team approach seemed to allow for increased collaboration, peer-peer learning, and an increased likelihood of project completion. For example, a longitudinal team-based curriculum allow residents to work as a team member as a junior resident before taking the lead on a multidisciplinary QI team as a senior resident.12 While the utilization of team-based work may tempt the possibility for multiple initiatives, involvement in more than 1 QI project was shown to cause a decline in project progress.20

The ideal timeline for a QIPS program is an important factor in curriculum development. There are very few evidence-based recommendations, and the small amount of literature that does exist offers conflicting findings. The University of Washington demonstrated overwhelming success in utilizing a longitudinal curriculum that allotted 9 months for project and curriculum completion32; a stark contrast to the success reported with the implementation of an intensive 5-week Obstetrics and Gynecology QI “bootcamp” curriculum.25 While more research on ideal curriculum length is needed, feedback from the ASCO Quality program suggested allotted time for QI project completion should be greater than 6 months.26 Regardless of curriculum length, a significant body of evidence supports the implementation of “protected time.” “Protected time” for residents allows for increased levels of didactic attendance, more meaningful participation, and positive program feedback.14,18,21,32 However, Stanford’s Neurology resident-driven QI curriculum presents conflicting findings to this growing body of evidence. Their curriculum was associated with a marked increase in QI project participation and scholarly output, despite the absence of scholarly requirements or protected project time.23

The ideal structure of the didactic component was another important component for curriculum development and exhibited in a myriad of ways. Several programs used the IHI Open Schools modules as centerpiece of their didactic structures.10,19,27 However, an asynchronous programs with a lack of oversight demonstrated low QIPS resident retention rates.29 The lack of oversight can be addressed by utilizing a chief quality resident, responsible for facilitation of resident participation in didactics and project completion.27 The overarching themes seems to be rooted in flexibility. Different specialties can utilize different means of QIPS education and components that best align with their schedule.15 In surgical training for example, morbidity and mortality conferences should be a cornerstone of QIPS curriculum.10

The breadth of literature focused on UME specialty-specific curriculum development was significantly smaller. The utilization of IHI modules again was a common finding, programs noting the future utility in a adopting a flipped classroom approach in improving QI knowledge.11,13 However, in evaluation of short-term (5-week) flipped classroom approach with a project-pitch component, less than 50% of students reported that they recommend this curriculum to fellow students.11

In summary, the 2 most frequently witnessed and reported elements for a successful curriculum structure involve the necessity for a combined didactic-experiential longitudinal structure and experienced QI faculty as session leads and project mentors. QI simulation demonstrated success in GME in both General- and Specialty-specific programs.9,30,50 Regardless of specialty, length or composition, structure should be flexible to fit the needs of the particular learning cohort.

Evaluation

Like QIPS curriculum structures, there is little standardization amongst programs seeking to evaluate program participants and curriculum success. Many programs use multiple methods of evaluation and given the smaller body of evidence (33 articles) compared to the 47 that discussed structure, the evaluation strategies were not further stratified into groups. Figure 5 demonstrates the frequency of the different evaluation mechanisms used.

F5
Figure 5.:
Curriculum evaluation strategies. The frequency of evaluation strategies used, demonstrated as a percentage of the 37 articles that included a methods of evaluation discussion.

The most frequent evaluation strategy used was the broad category of program-specific pre- and postcurriculum surveys. Figure 5 demonstrates 70.3% of program used some sort of nonstandardized or custom survey to best evaluate their participants. The content and timing of these surveys is an important factor to best describe the goals of the surveys. Most studies administered surveys immediately before and after program completion, with a large majority of those focusing their evaluation on assessment of QI knowledge domains, QI methodologies, and confidence in QI.11,18–20,26,27,37,49,53 Most study structured their evaluation utilizing a Likert scale, the most widely used approach to scaling response in survey research. Other studies with similar focus utilized non-Likert evaluation, electing to gather feedback in a more free-form manner.25 Two studies focused on similar competencies as the studies above, but sought to evaluate long term retention administering studies 1 year and 18 months after program completion, respectively.17,23 Although QI knowledge and confidence were the sole focus of most evaluations, many studies also included supplemental evaluation metrics. Some examples included prior training and curriculum goals,33 assessment and barriers to conduct QI work,42 and gained communication skills with multidisciplinary teams.55 Of the remaining studies the second most common focus involved surveys assessing participants evaluation and satisfaction of the curriculum, the majority of these surveys again using Likert scale structures.22,34,35,41,45 A few studies developed evaluation with a largely different focus, those being quality event reporting,14 patient safety competencies,29 and resistance to change.15 The significant deviation was likely a result of curriculums with an exclusive focus on patient safety or slightly different scope than traditional QIPS programs.

The problem with the development of curriculum specific surveys is the lack of validity and generalizability of outcomes. Developing programs should seek to utilize a standardized, validated QIPS assessment tool that allows for reliable curriculum comparison. The current gold standard for QI assessment tools is the Quality Improvement Knowledge Application Tool (QIKAT) and as of 2014, the Quality Improvement Knowledge Application Tool – Revised (QIKAT – R): a widely adopted, reliable, and validated case based assessment of QI knowledge and applications.56 Adoption of this standardized evaluation and others (ie, AQIKS) are increasingly being utilized. Figure 5 demonstrates 24.3% of studies utilized a “Reliable QI Evaluation Tool” (which includes QIKAT, QIKAT-R, AQIKS30). The use of the QIKAT-R tool has been utilized in both a long-term51 and immediate45 postcourse UME scenario, in an interdisciplinary GME curriculum,53 and in specialty-specific curriculums (ie, internal medicine21 and psychiatry33). Furthermore, in addition to the old QIKAT being successfully adopted in a surgical35 quality curriculum, the widespread applicability and utilization across the UME and GME QIPS education window is demonstrated. These highly reliable and standardized evaluation tools (and the QIKAT-R in particular) will allow for a validated and generalizable pool of data moving forward which enables more detailed comparisons between extremely diverse QIPS programs.

Other commonly utilized methods of evaluation were scholarly activity, which was seen in 29.7% of the included articles and qualitative or descriptive feedback from participants, which was seen in 8.1% of studies as demonstrated in Figure 5. “Scholarly Activity Evaluation” encompassed both project evaluations, participation QIPS academic activity, and career outcomes. These metrics allow for a relatively easily collectible avenue to evaluate both long and short-term influence of the QIPS curriculum on participants. For example, a family medicine curriculum evaluated academic productivity of participants over a 10 year span from 2008-2018, in which yearly-resident led scholarship products were tracked,20 while other programs evaluated the QIPS research production of residents from before and after program implementation.19 While both mechanisms of evaluation provide an alternative quantitative tool to measure program success, QIPS programs’ value on career outcomes can be measured through tracking participant job placement and roles.52 Qualitative and descriptive feedback in contrast, encompassed a more descriptive and explorative approach to program evaluations. Here, participants provided free-form feedback that offered qualitative data specific to their program.12,29

As demonstrated above, each method of evaluation provides a unique benefit when evaluating the success of a QIPS program. The authors’ recommendations parallel that of the 2019 guide to developing a QIPS evaluation program. Establishment of a timeline and a mixed methods evaluation design (that utilizes multiple sources of quantitative and qualitative data) is most effective.43 Utilization of standardized assessment tools, specifically the QIKAT-R, can provide a standardized, quantitative body of data that will allow for a direct short-term comparison to other programs. Similarly, alumni tracking of QIPS job roles and scholarly activity can provide quantitative long-term program effectiveness.43 The addition of qualitative and descriptive surveys or feedback platforms can supplement these quantitative findings and provide an excellent source of qualitative data.

Outcomes

Overall, most studies reported widespread success with implementation of their curriculums. Here, the authors first compare the programs that utilized the QIKAT-R, the gold standard for standardized QI knowledge assessment. The target populations and program specialties that utilized QIKAT-R varied significantly but nearly all reported significant positive QIKAT-R score improvement. Highlights included a psychiatry resident curriculum demonstrating the importance of resident engagement in project development and confident data analysis to be a particular driver in QIKAT-R score improvements.33 An internal medicine-driven curriculum focused on regular QI education session, protected time, and integrated longitudinal project to be key factors in significant score improvement.21 The scale proved successful in UME as well, as participants demonstrated significant improvement in scores after year one of the UME curriculum, while positive trends in scores from year 2 onward demonstrate the potential value for spaced repetition and continuous QIPS training.45 The multispecialty QIPS program for fellows attributed their non-significant QIKAT-R score improvement to a curriculum leadership skills focus that may have not been adequately assessed by the tool.53 For future curriculum design and education, curriculum objectives should be developed to align with QIKAT-R evaluation metrics, effectively allowing for standardized and generalizable comparisons of different programs.

Overwhelmingly, the most frequently utilized tool used to evaluate both participants and programs involved pre- and postprogram surveys. Surveys were usually structured using a Likert scale and varied in the timeline administered, structure, and focus. However, many had similarities in the competency of focus, QIPS knowledge, and confidence being the most common. Most programs reported significant improvement in self-assessed QI knowledge.12,14,17,19,25,26,40,53 More specifically, knowledge domains and competencies included QIPS methodologies,25 project development and system thinking,19 root cause analysis and patient safety,53 among others. Significant improvement in self-reported QIPS confidence was also frequently demonstrated.11,17,18,23,26,40 Outcomes were often mixed: a family medicine clerkship curriculum, for example, demonstrated both significant improvement in QIPS knowledge and confidence, but also reported that less than 50% of participants would recommend the curriculum for future students.11 An important takeaway is highlighted from this finding: while curriculums may report progress in improving QIPS knowledge, evaluators must also consider qualitative data to ensure positive user-based feedback. Other competency areas that reported significant improvement involved comfort with QIPS17,37 and project development skills (which included but were not limited to systems thinking, QI principles, challenges, designing/writing problem statements, leading QI initiatives).19,25,35,37 The most resoundingly positive program feedback came through the 9-month QIPS curriculum for PGY-3 psychiatry residents that included protected didactics and protected project development time. Here, 96.1% (74/77) of participants felt the curriculum had a positive impact on their learning and believe their clinical practice would change as a result.32

Beyond the validated QIKAT evaluation and curriculum specific surveys, scholarly output followed as a common outcome measurement tool. From a scholastic evaluation standpoint, academic productivity was highlighted as an effective way to evaluate short-term curriculum influence while career outcomes (alumni tracking) provided evidence for long-term program effectiveness.43 Alumni tracking was used to demonstrate the long-term effectiveness of the dedicated 2-year QIPS fellowship—96% of program graduates entered academic positions that involved QIPS administration, research, or education.52 While undergraduate and more general QIPS program may not yield the same influence on career outcomes, there is strong evidence for increased academic productivity across a spectrum of different GME specialties including family medicine,12 internal medicine,19 and pathology27 as a result of program participation.

Finally, descriptive feedback from participants yields important qualitative data from the lens of the learner and can be referenced for future program development. These unique points of data are even more valuable as participant feedback can provide information regarding program components specific to different curriculums. An example is taken from a 10-month Internal Medicine curriculum: focus groups endorsed a low retention of information and skills beyond curriculum sessions and attributed this to 5-week gaps between didactic sessions and lack of protected project/didactic time.18 Different programs, such as the QIPS curriculum for Psychiatry residents at the University of Wisconsin, documented residents’ satisfaction with the structure of the longitudinal QI rotation, supporting their believe that this was the most effective way to accomplish meaningful QIPS work.32 Other qualitative user feedback included beliefs that engagement in QIPS project design was crucial to gaining QI skills,33 while others suggested that there may be value in less-time intensive curriculums (compared to the year long QIPS fellowship) to better meet QIPS needs.53

Conclusion

QIPS programs in both UME and GME education have showcased an overwhelming, strongly positive influence on medical students and residents extraclinical education. Regardless of specialty and structure, most curriculums demonstrated success in increasing knowledge and confidence in QIPS, led to an increased academic productivity in the QIPS domain, and had an influence on career outcomes of both medical students and residents. Sadly, there remains no allopathic medical college with a mandatory curriculum in QIPS. Future policy should seek to prioritize this learning topic and future education administrators should seek to standardize curriculum development. Longitudinal programs with a reliance on both didactic and experiential learning, faculty oversight with expertise in QIPS (via accredited QIPS programs), protected time, and standardized evaluations are key components for all future QIPS curriculums.

Moving forward, further research that seeks to evaluate ideal program length, structure, and competencies of focus is paramount. The lack of evidence in these areas significantly inhibits the standardization of QIPS curriculums across the country. This review also demonstrates the growing standardized body of short-term quantitative outcomes (QIKAT-R), scarcity of long-term knowledge retention outcomes, and frequent lack of enthusiasm among residents. These findings suggest that a follow-up systematic review comparing short term QI knowledge may be a valuable follow-up contribution to the literature, while future studies should seek to explore long-term QI retention and better understand the lack of enthusiasm among residents for recommending these curriculums to their peers.

Author Contributions

Mr Li and Dr Nash developed the scope of the article, defined search criteria databases for use. Mr Li reviewed the prevailing literature and completed the preliminary review of the literature which was later reviewed by Dr Nash. Mr Li drafted the article with Dr Nash’s consistent input and revisions. All authors contributed to the interpretation of literature and made critical revisions to the article.

Conflicts of Interest

The authors have no conflicts of interest to disclose.

Funding

The authors have no funding disclosers to report.

References

1. Institute of Medicine Committee on Quality of Health Care in A, Kohn LT, Corrigan JM, Donaldson MS. To Err is Human: Building a Safer Health System. National Academies Press (US) Copyright 2000 by the National Academy of Sciences. All rights reserved; 2000.
2. Institute of Medicine Committee on Quality of Health Care in A. Crossing the Quality Chasm: A New Health System for the 21st Century. National Academies Press (US) Copyright 2001 by the National Academy of Sciences. All rights reserved.; 2001.
3. Weiss KB, Bagian JP, Wagner R. CLER pathways to excellence: expectations for an optimal clinical learning environment (executive summary). Journal of Graduate Medical Education 2014;6:610–611.
4. Cheng MKW, Collins S, Baron RB, et al. Analysis of the interprofessional clinical learning environment for quality improvement and patient safety from perspectives of interprofessional teams. J Grad Med Educ 2021;13:822–832.
5. McGlynn EA. Improving the quality of U.S. health care—what will it take? N Engl J Med 2020;383:801–803.
6. Walton M, Woodward H, Van Staalduinen S, et al. The WHO patient safety curriculum guide for medical schools. Qual Saf Health Care 2010;19:542–546.
7. Oglesby WH, Hall AG, Valenta AL, et al. Accrediting graduate programs in healthcare quality and safety. Am J Med Qual 2021;36:441–448.
8. Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med 2018;169:467–473.
9. Harbell MW, Methangkool E. Patient safety education in anesthesia: current state and future directions. Curr Opin Anaesthesiol 2021;34:720–725.
10. Bowe SN, McCormick ME. Resident and fellow engagement in safety and quality. Otolaryngol Clin North Am 2019;52:55–62.
11. Mills GD, Kelly S, Crittendon D, et al. Evaluation of a quality improvement experience for family medicine clerkship students. Fam Med 2021;53:882–885.
12. Pohl SD, Van Hala S, Ose D, et al. A longitudinal curriculum for quality improvement, leadership experience, and scholarship in a family medicine residency program. Fam Med 2020;52:570–575.
13. Shaheen AW, Fedoriw KB, Khachaturyan S, et al. Students adding value: improving patient care measures while learning valuable population health skills. Am J Med Qual 2020;35:70–78.
14. Arnell M, Demet R, Vaclavik L, et al. Use of a rubric to improve the quality of internal medicine resident event reporting. MedEdPORTAL 2021;17:11189.
15. Buckley R, Spadaro A, Rosin R, et al. Comparing the effects of design thinking and a3 problem-solving on resident attitudes toward systems change. J Grad Med Educ 2021;13:231–239.
16. Cohen E, Bradley J, van Aalst R, et al. Evolution of a resident quality improvement curriculum: lessons learned on the path from innovation through stability to contraction. Jt Comm J Qual Patient Saf 2019;45:575–579.
17. Durstenfeld MS, Statman S, Carney K, et al. Swimming with sharks: teaching residents value-based medicine and quality improvement through resident-pitched projects. J Grad Med Educ 2020;12:320–326.
18. Kennedy AG, Burnett M, Muthukrishnan P, et al. “I Think i was losing the forest for the trees”: Evaluation of an internal medicine residency quality improvement curriculum. Medical Science Educator 2020;30:197–202.
19. Mustafa RA, Gillenwater K, Miller SK, et al. QIPS CURE: Implementing a quality improvement and patient safety curriculum and resident experience. J Grad Med Educ 2020;12:469–477.
20. Parekh N, Lebduska E, Hoffman E, et al. A longitudinal ambulatory quality improvement curriculum that aligns resident education with patient outcomes: a 3-year experience. Am J Med Qual 2020;35:242–251.
21. Baxter K, Petz C, Middleton JL, et al. Re-thinking how we teach quality improvement: adding meaning to an ACGME requirement. J Gen Intern Med 2019;34:1662–1667.
22. McInnis RP, Lee AJ, Schwartz B, et al. A quality improvement curriculum for the neurology clerkship: a practice-based approach to discharge education. eNeurologicalSci 2019;16:100196.
23. Miller-Kuhlmann R, Kraler L, Bozinov N, et al. Education research: a novel resident-driven neurology quality improvement curriculum. Neurology 2020;94:137–142.
24. Abbott JF, Pradhan A, Buery-Joyner S, et al. To the point: integrating patient safety education into the obstetrics and gynecology undergraduate curriculum. Journal of Patient Safety 2020;16:E39–E45.
25. Goldfarb IT, Smith NA. QI Bootcamp: feasibility and acceptability of a novel approach to training residents in process improvement. Am J Obstet Gynecol 2019;221:656–658.
26. Keng M, Quinn D, Cunningham G, et al. ASCO quality training program: a five-year review. JCO Oncol Pract 2020;16:e1243–e1248.
27. Sapatnekar S, Demkowicz R, Chute DJ. Implementation of a quality and patient safety curriculum for pathology residency training. Academic Pathology 2021;8:2374289521998046.
28. Carney KP, Topoz I, Vu TT, et al. Development of a quality improvement curriculum in a pediatric emergency medicine fellowship. Pediatr Emerg Care 2019;35:373–376.
29. Stankovic C, Wolff M, Chang TP, et al. A national patient safety curriculum in pediatric emergency medicine. Pediatr Emerg Care 2019;35:519–521.
30. Greenlaw C, Jacob S, Cheston CC. Pediatric quality improvement (QI) virtual practicum: adapting a QI simulator. MedEdPORTAL 2020;16:10929.
31. Grover P, Volshteyn O, Carr DB. Physical medicine and rehabilitation residency quality improvement and research curriculum: design and implementation. Am J Phys Med Rehabil 2021;100(2S Suppl 1):S23–S29.
32. Reardon CL, Hafer R, Langheim FJP, et al. A quality improvement curriculum for psychiatry residents. MedEdPORTAL 2020;16:10870.
33. Ridout SJ, Ridout KK, Theyel B, et al. A novel experiential quality improvement training program during residency improves quality improvement confidence and knowledge: a prospective cohort study. Acad Psychiatry 2020;44:267–271.
34. Kurup AN, Rosier AS, Henrichsen TL, et al. Implementing a radiology residency quality curriculum to develop physician leaders and increase value for patients. Radiographics 2020;40:505–514.
35. Ahuja V, Gorecka J, Yoo P, et al. A longitudinal course pilot to improve surgical resident acquisition of quality improvement skills. PLoS One 2021;16:e0254922.
36. Berger ER, Kreutzer L, Halverson A, et al. Evaluation of changes in quality improvement knowledge following a formal educational curriculum within a statewide learning collaborative. J Surg Educ 2020;77:1534–1541.
37. Colvin J, Feng X, Lipman J, et al. QIC: A pilot quality improvement curriculum utilizing interactive, team-based workshops. J Surg Res 2021;261:167–172.
38. Kelz RR, Sellers MM, Merkow R, et al. Defining the content for a quality and safety in surgery curriculum using a nominal group technique. J Surg Educ 2019;76:795–801.
39. Papa FJ, Alexander JH. Aggregated student confidence estimates support continuous quality improvements in a competencies-oriented curriculum. BMJ Open Quality 2019;8:bmjoq–b2018.
40. Schroll R, Paramesh A, Guidry C, et al. A structured quality improvement educational curriculum increases surgical resident involvement in QI processes. J Surg Educ 2020;77:e183–e186.
41. Vaughn NH, Hassenbein SE, Black KP, et al. Important elements in the quality improvement curriculum for orthopaedic residents. J Bone Joint Surg Am 2019;101:e28.
42. Aboumrad M, Carluzzo K, Lypson M, et al. Career impact of the chief resident in quality and safety training program: an alumni evaluation. Acad Med 2020;95:269–274.
43. Butcher RL, Carluzzo KL, Watts BV, et al. A guide to evaluation of quality improvement and patient safety educational programs: lessons from the va chief resident in quality and safety program. Am J Med Qual 2019;34:251–259.
44. Cruz MS D de la, Smith RS, Silverio AE, et al. What we learned in the development of a third-year medical student curricular project. Perspectives on Medical Education 2021;10:167–170.
45. Dumenco L, Monteiro K, George P, et al. Outcomes of a longitudinal quality improvement and patient safety preclerkship curriculum. Acad Med 2019;94:1980–1987.
46. Genuardi FJ, Kelly MN, Rosenberg EI, et al. Teaching students how to improve safety and quality in two children’s hospitals: building a pediatric clerkship patient safety and quality experience. Acad Pediatr 2019;19:712–715.
47. Gottenborg E, Anstett T, Diaz M, et al. Teaching students to transform: The health innovations scholars program 6 years later. Am J Med Qual 2020;36:277–280.
48. Harbell MW, Donnelly M, Rastogi R, et al. Impacting the next generation: teaching quality and patient safety. Int Anesthesiol Clin 2019;57:146–157.
49. Hommel E, Sonstein L, Raji M. Feasibility and effectiveness of a quality improvement curriculum for combined medicine subspecialty fellows. Am J Med Qual 2022;37:137–144.
50. Huang IA, Simon W, Lu Y, et al. Simulation use in quality improvement and patient safety training for residents. Am J Med Qual 2021;36:371.
51. Murphy TJ, Saldivar BN, Holland CK, et al. A quality education: a comprehensive review of a combined longitudinal and specialty track quality improvement and patient safety medical school curriculum. Am J Med Qual 2022;37:32–38.
52. Myers JS, Lane-Fall MB, Perfetti AR, et al. Demonstrating the value of postgraduate fellowships for physicians in quality improvement and patient safety. BMJ Qual Saf 2020;29:645–654.
53. Neumeier A, Levy AE, Gottenborg E, et al. Expanding training in quality improvement and patient safety through a multispecialty graduate medical education curriculum designed for fellows. MedEdPORTAL 2020;16:11064.
54. Olsen G, Quam J, Wolfe D, et al. Quality improvement education: redesigning intermountain healthcare’s advanced training program for a value-based learning health care system. Qual Manag Health Care 2021;30:209–211.
55. Patel A, Adili H, Rojas JC, et al. Integrating physicians into lean quality improvement through a structured educational program: the ECLIPSE program. Am J Med Qual 2022;37:6–13.
56. Singh MK, Ogrinc G, Cox KR, et al. The quality improvement knowledge application tool revised (QIKAT-R). Acad Med 2014;89:1386–1391.
Keywords:

quality improvement; patient safety; medical education; curriculum

© 2022 the American College of Medical Quality