The quality improvement (QI) and patient safety (PS) movements in recent years have had important implications for undergraduate and graduate medical education, including focused attention on duty hours reductions,1–4 appropriate supervision of trainees,4,5 and communication during resident hand-offs.6 Furthermore, there is a growing imperative to teach QI and PS in medical education.
The Association of American Medical Colleges now endorses the introduction of formal QI education across the medical education continuum, spanning undergraduate, postgraduate, and continuing medical education levels.7,8 Both the Accreditation Council for Graduate Medical Education (ACGME)9 and CanMEDS10,11 competency frameworks define essential physician competencies that relate to quality and safety. These developments coincide with the recognition that engagement in QI represents an emerging career path for clinicians.12
A previous systematic review of QI education for clinicians found that most curricula demonstrated improvement in learners' knowledge by applying sound adult learning principles.13 However, only 10 of the 39 studies targeted trainees (of which only 2 involved medical students). Given the increasing recognition of the need to teach QI to students, we systematically reviewed published curricula in QI or PS specifically directed at medical students or resident trainees. We sought to (1) describe their educational content and teaching methods used, (2) assess the learning outcomes achieved, and (3) determine factors that promoted or limited the successful implementation of these curricula.
Method
Literature search
We searched for relevant English-language studies from January 1, 2000 to January 2009 using electronic literature databases including Medline, EMBASE, and HealthSTAR. We chose January 2000 as the cut-off of the search period in order to capture articles describing educational efforts that arose following the release of the Institute of Medicine's report, To Err Is Human,14 which initiated the current widespread interest in QI and PS.
The search strategies combined Medical Subject Headings terms and text words related to QI and PS (e.g., medical errors, safety, quality assurance) with those related to undergraduate and graduate medical education. (We will be glad to send interested readers our search strategy.) We also hand-searched bibliographies of all included reports and relevant review articles.
Eligibility criteria
We included any article that, at a minimum, described a curriculum that explicitly identified its goal as exposing medical students or residents to concepts in QI or PS and that outlined specific teaching methods used to achieve this educational goal. We identified concepts in QI or PS by screening for descriptors in the article's title or text, including general terms such as quality improvement, systems learning, systems-based practice, patient safety, as well as specific topics widely recognized as falling within the domains of QI (i.e., continuous QI, audit and feedback, process mapping, change management) and PS (i.e., systems thinking, root cause analysis, human factors engineering, incident reporting, dealing with errors, safety culture, error disclosure).
Studies were excluded if they (1) primarily focused on practicing clinicians (e.g., if the intervention targeted members of an academic clinic, some of whom happened to be trainees), (2) were predominantly QI interventions that happened to be delivered in training settings (e.g., an audit and feedback intervention delivered in a resident clinic), or (3) described curricula in which QI or PS topics were included as a minor component of a larger curriculum (e.g., a single lecture on QI in a broad curriculum on managed care).
Article review process
Two of us (B.W. and K.S.) independently reviewed titles and abstracts to identify eligible articles. When in doubt, the full text of each article was obtained to finalize article inclusion or exclusion. The article screening process was followed by independent abstraction (B.W. abstracted data from all included articles; K.S. and E.E. carried out duplicate data abstraction independently) using a structured data entry form. Disagreements at both the article screening and data abstraction stages were resolved by consensus, involving a third reviewer if necessary.
Consistent with the Best Evidence Medical Education (BEME) review protocol,15 we extracted curricular descriptors as well as key methodological features for those articles that included an evaluative component. We classified learning outcomes using Kirkpatrick's model,16 which includes impacts on learners' satisfaction (Level 1), changes in learners' attitudes (Level 2A), measures of learners' knowledge and skills (Level 2B), changes in learners' behavior (Level 3), changes to clinical processes (Level 4A), and benefits to patients (Level 4B).
We supplemented the structured data abstraction with a detailed thematic analysis of each article's text to identify factors that the authors regarded as promoting or limiting curricular implementation. One of us (B.W.) established a framework for key factors that influenced curricular implementation based on an initial detailed reading of all included studies as well as existing literature on curricular development and implementation in medical education more generally.17 After iterative review and modification by other investigators, we independently applied the final framework to code each study (B.W. coded all studies; K.S. and A.K. carried out duplicate coding independently).
Assessment of study quality
For studies with an evaluative component, we assessed the strength of the findings using a modified version of the BEME protocol.15 The BEME rating system for strength of findings assigns a rating of Level 1 when no clear conclusions can be drawn, Level 2 when results are ambiguous but exhibit a trend, Level 3 when conclusions can probably be based on the results, Level 4 when results are clear and very likely to be true, and Level 5 when results are unequivocal. Although widely used, the BEME protocol does not include explicit features to guide these judgments. We therefore adopted the protocol to inform ratings of the strength of study findings using considerations of sample size, number of sites, study design, completeness of data, and response rate.
Analysis
We anticipated substantial heterogeneity of study design and reported outcomes, so we chose not to pursue quantitative synthesis. We summarized educational content, teaching methods, and learning outcomes (for studies with an evaluative component only) using simple descriptive statistics. We included themes identified by the detailed thematic analysis if they were observed in two independent sources. We summarized these themes and highlighted key excerpts that illustrate these themes to describe important factors that limited or promoted implementation of QI and PS curricula.
All data were previously published and publicly available. Therefore, our study did not meet criteria for submission to the local institutional review board for ethical approval.
Results
Characteristics of included curricula
Of 953 citations identified by the electronic search, 41 curricula met eligibility criteria (see Figure 1),18–58 27 (66%) of which provided a curricular description along with some form of evaluation.18–44 The vast majority (38; 93%) of reports came from U.S. training programs; of the remaining three, two (5%) came from Canada47,58 and one (2%) from the United Kingdom.37 Participating learners consisted of medical students in 14 studies (34%), residents in 24 (59%), and both in 3 (7%). Curricula for residents primarily came from internal medicine (58%) and family medicine (21%) training programs. Twenty-five (61%) of the curricula for students and residents were mandatory.
;)
Figure 1:
Literature search and study selection process for identifying quality improvement and patient safety curricula published between 2000 and 2008.
Curricular features
The curricula addressed a range of QI and PS content (see Table 1 and Table 2), but the most common topics consisted of continuous QI (21 studies, 51%), root cause analysis (17 studies, 41%), and systems thinking (16 studies, 39%). Most curricula combined didactic and experiential learning; detailed case discussions and Web-based learning were less frequently used.
Table 1: Features of 41 Quality of Care and Patient Safety Curricula Described in Studies Published Between 2000 and January 2009
Table 2: Characteristics, Teaching Methods, Educational Content, and Learning Outcomes of 41 Curricula in Quality Improvement or Patient Safety for Trainees Described in Studies Published Between 2000 and January 2009
Table 2: (Continued)
Table 2: (Continued)
Table 2: (Continued)
Table 2: (Continued)
Table 2: (Continued)
Table 2: (Continued)
Among curricula targeting medical students, seven targeted preclinical medical students and seven targeted clinical medical students. Some curricula were integrated into one course or rotation, whereas others were delivered as stand-alone sessions. The majority of these curricula generally involved fewer than 10 contact hours, often consisting of a single session. Five curricula involved medical students in QI or PS projects.
QI and PS curricula targeting residents were similarly brief (approximately 10 contact hours) but more often involved multiple encounters (i.e., 2–5). Approximately half of the curricula incorporated their content into existing core rotations; others occurred as stand-alone sessions or elective rotations. All curricula for residents were delivered in clinical settings (e.g., ambulatory clinic or inpatient teaching unit). Residents participated in QI or PS projects in 14 (58%) of the curricula.
Study designs and outcomes
Table 2 summarize the outcomes, designs, and main results of the 27 studies that included an evaluative component (a more detailed summary of the study outcomes is provided in the Appendix, which can be accessed at https://links.lww.com/ACADMED/A20). The most common design was a simple before–after comparison (11; 42%). Five (19%)28,29,34,35,38 evaluations included a contemporaneous control, and two of these used a randomized design.29,35 One of these randomized, controlled studies evaluated a curriculum implemented at seven U.S. training programs,29 and the other evaluated programs at 18 U.S. teaching hospitals.38 However, most studies (24; 92%) came from single centers and had methodological concerns that undermined the results, such as low response rates and small sample sizes (median 41 participants; interquartile range 20–106).
Evaluations of curricula targeting medical students primarily measured learners' knowledge, with a lesser emphasis on behavior change. Only one medical student curriculum targeted changes in clinical processes.49 Curricula for residents more commonly involved residents in QI projects (14; 58%) and frequently reported outcomes that measured improvements in process of care. Only two studies reported benefits to patients.24,28
Table 3 reports Kirkpatrick learning outcomes by training level. The following section summarizes each learning outcome in greater detail.
Table 3: Kirkpatrick Learning Outcomes, by Trainee Level, of 27 Studies of Quality and Safety Curricula That Had an Evaluation Component, 2000 to January 2009
Learners' satisfaction
Satisfaction was usually measured on a Likert scale from poor to excellent. The majority of learners were satisfied with the QI curricula, consistently rating the curricula as relevant and useful. Only two studies reported low satisfaction ratings. One was conducted with first-year medical students35 and reported early termination of the study due to learners' dissatisfaction with the curriculum. The other study involved second-year medical students.24 Students participating in this curriculum also expressed a number of concerns, including skepticism about the project being an efficient use of time. Among their concerns, 84% of students reported dissatisfaction with the chart audit exercise.
Learners' attitudes
Learners generally exhibited positive attitudes prior to exposure to the curricula. For instance, the majority of learners already regarded QI and PS as important topics relevant to future practice. Given these positive baseline attitudes, most curricula reported minimal impacts on attitudinal outcomes.
Knowledge acquisition
Acquisition of curricular content was usually assessed using tests of knowledge designed by study teams, though some studies used established assessment tools such as the Quality Improvement Knowledge Assessment Tool.59 With self-assessed knowledge outcomes, learners generally rated their knowledge highly and improved from baseline. All eight studies that quantified knowledge acquisition reported statistically significant improvements.
Behavioral change
The five studies21,26,30,37,43 that reported behavioral changes all used self-reported outcomes. Nonetheless, only two studies suggested any improvements in the behaviors targeted by the curricula. One study reported that although many students had disclosed errors to a peer (71%) or faculty member (46%), only 7% had ever used a Web-based reporting system highlighted in the curriculum.30 The other study that reported an outcome related to learners' behavior targeted disclosure of medical errors to patients and found that only 7% of learners reported having made such a disclosure following exposure to the curriculum.26
Changes in clinical processes
A number of studies involved chart audits, modified morbidity and mortality conferences, or participation in a QI project. Seven of the 13 studies (54%) reported significant improvements in processes of care,20,24,28,31,36,41,43 including increased microalbuminuria screening,20 documentation of foot and eye examinations,24 and increased monofilament testing28 for patients with diabetes, increased screening for elevated body mass index in an ambulatory internal medicine clinic,36 reduction in inappropriate telemetry use on an inpatient medical service,43 increased discharge dictations with complete medication information,41 and increased immunizations in a pediatric clinic.31
Benefits to patients
Two studies measured benefits to patients in terms of intermediate clinical outcomes (serum HbA1c in both cases). In one study, 13 internal medicine residents performed chart audits on patients with diabetes and reflected on solutions to identified problems.28 That Hstudy reported a decrease in HbA1c levels of 0.4% for patients cared for by participants in the intervention group compared with an increase of 0.7% in the control group (P < .001). The other study, in which 77 second-year medical students audited charts for patients with diabetes,24 reported a reduction in HbA1c levels from 7.7% before implementation of the QI curriculum to 7.2% afterwards (P < .001).
Factors that influenced curricular implementation
Of the 41 included reports, 34 (83%) described factors that influenced implementation of QI and PS curricula (see Table 4). Commonly cited barriers related to learners included competing educational demands and the level of initial buy-in or enthusiasm. For faculty, many reports highlighted the problem of inadequate numbers of teachers with requisite expertise, and the time commitment required for those few faculty members (often only one or two at a given institution with such expertise). Barriers related to the curricula themselves included achieving the appropriate balance of didactic and experiential learning and scheduling the curriculum amidst existing classes and rotations. Important aspects of the learning environment included the institutional culture with respect to quality and safety, hospital operational support (e.g., some authors noted the positive impact on trainees of including hospital executives or faculty role models involved in local improvement efforts), as well as the availability of information systems that could facilitate QI projects undertaken by trainees.
Table 4: Factors That Influence the Successful Implementation of Quality of Care or Patient Safety Curricula, From 34 Studies Published Between 2000 and January 2009
Table 4: (Continued)
Many of the same implementation issues emerged across all curricula irrespective of learner level (i.e., undergraduate or postgraduate). However, some factors were more commonly cited as important factors only for curricula targeting residents (e.g., time pressures and the need for ongoing financial, educational, institutional, and operational support), perhaps because of the greater inclusion of QI or PS projects in curricula for residents. A barrier unique to curricula that targeted medical students in the preclinical years was their perception of the unimportance of the material compared with traditional clinical content.
Discussion
We identified 41 QI and PS curricula that specifically targeted medical students or residents. Concepts of continuous QI, systems thinking, and root cause analysis constituted the most common topics covered, and specific projects undertaken often involved chart audits. Despite the heterogeneity in educational content and teaching methods, most curricula were well accepted and led to learners' knowledge acquisition. Resident involvement in experiential QI projects such as chart audits also frequently led to significant improvements in processes of care.
Few studies demonstrated changes in learners' behavior or potential patient benefits. Although some reports suggest that educational interventions have the potential to change behavior or improve health outcomes, most studies lack good-quality evidence to support their findings.60 There are examples of well-designed continuing medical education interventions that are sequenced and make use of interactive techniques that lead to changes in learners' behaviors and health outcomes. However, those studies often centered on screening, smoking cessation, and communication skills and may not translate to more complex curricular content areas, such as QI and PS.61–63 In fact, for QI and PS, improving patient outcomes as a result of educational efforts represents a particularly daunting task, given that intensive, large-scale QI efforts often fail to demonstrate improvements in health outcomes.64,65 Also for some tools of QI and PS, including ones that commonly appeared in the curricula we reviewed (e.g., root cause analysis), little empiric evidence guides recommendations on how to design or use these tools.65 Consequently, even with optimal delivery of the target educational content, the degree to which organizational or patient outcomes might improve remains unclear.
Our results complement those of a systematic review of educational efforts in QI for clinicians in general13 in that well-established adult learning techniques (e.g., experiential learning) were identified as key factors for success in delivering curricula in QI and PS. However, our review, which included 34 newer reports of curricula specifically targeting trainees, demonstrated that residents' involvement in QI and PS curricula can lead to meaningful improvements in clinical processes, a novel finding compared with those of the previous review.
Our review also identified important barriers and facilitators to implementation that are likely unique to curricula in the undergraduate and postgraduate settings. Many of the studies identified barriers commonly encountered with new curricular initiatives in general.17 For example, most of the curricula relied on small numbers of faculty members with a personal interest in QI or PS to teach the curriculum, often resulting in burdensome time commitments. Many reports highlighted the need for greater faculty development to achieve sufficient numbers of teachers of QI and PS topics for both medical student and resident curricula. Some curricula addressed these issues by developing teaching materials that circumvented the need to have faculty experienced in QI or PS.36
Competing educational demands and achievement of learner buy-in also represented major issues for curricula at all levels. However, the only two reports24,35 that noted these as potentially insurmountable obstacles were ones that targeted medical students at preclinical stages. Learners reported significant dissatisfaction with key elements of the curricula, which suggests that clinical experience represents a prerequisite for appreciating the importance and relevance of QI or PS concepts.
Curricula that targeted residents may require special consideration, perhaps because such curricula more commonly involved the learners in experiential projects, adding to time pressures and increasing the need for supporting infrastructure. Many residents did not complete their projects because of time constraints. Some programs addressed this problem by scheduling their curricula during less busy clinical rotations or research years.19 Having adequate personnel, financial, and technological resources to support curricula involving experiential projects was also important. For example, studies that made use of chart audits required administrative support to retrieve charts. Also, many QI projects depended on efficient availability of clinical data through information systems to determine whether improvements occurred.
Finally, a number of studies emphasized the importance of a local “safety culture,” substantially enhancing the curricular success when present and undermining it when absent. Other curricula that target nonmedical competencies (e.g., professionalism) also highlight the importance of the so-called hidden curriculum, where there is a discrepancy between the concepts trainees learn in formal educational venues and what trainees observe when supervised by attending staff in routine clinical practice.66–68 Preparing trainees for the fact that behavior of faculty in routine practice, design of the delivery systems in which they work, and institutional culture may not conform to accepted principles of QI and PS may reduce the discomfort reported by participants in some of the curricula we reviewed.
Our systematic review had several limitations. The literature examining the effectiveness of educational interventions in QI and PS exhibited substantial heterogeneity in terms of the content delivered, educational methods used, learners targeted, and learning outcomes reported. Also, many curricular evaluations involved weak study designs, occurred in single centers, had small numbers of learners, and often exhibited other methodological concerns. Consequently, we did not regard quantitative synthesis as appropriate.
Our thematic textual analysis of all of the included curricular reports identified a number of potentially important factors that promote or hinder implementation efforts. However, most of those reports did not have the identification of facilitators and barriers to implementation as their primary aim. Consequently, authors may not have recognized or reported aspects of the curricular implementation in a systematic fashion. Moreover, the vast majority of reports did not comment on the degree to which curricula had been sustained.
Conclusions
Improving the quality and safety of patient care has gained widespread acceptance as a central activity for the health care system. Clinicians will be expected to have acquired core concepts in QI and PS in order to apply them to improve their personal practices and help support institutional improvement efforts. Consequently, a consensus has emerged that QI and PS should be broadly taught to trainees, with ACGME9 and CanMEDS10 mandating such education and some students actively requesting it.69 Despite this emerging consensus, in 2006, few medical schools in the United States and Canada reported having explicit curricula in QI and PS,70 although it is likely that now (2010) there are more schools with such curricula.
The existing literature indicates that educational curricula focused on QI and PS are generally well accepted by trainees, effectively improve knowledge in these domains, and can even lead to important improvements in processes of care. Programs undertaking the development of curricula in QI or PS must recognize the significant time pressures and competing educational demands for trainees, as well as the requirements for adequate numbers of faculty with appropriate expertise and support for their contributions. To succeed, these curricula require engagement of educational and organizational stakeholders to promote adoption. Future research must better characterize the learner, faculty, and institutional factors that facilitate or hinder the promotion of sustained educational efforts focused on QI and PS for medical students and postgraduate trainees.
Funding/Support:
Dr. Wong received an honorarium from the Association of the Faculties of Medicine of Canada to write an earlier version of this review that appeared as a chapter in a monograph entitled “The Future of Medical Education in Canada.” Dr. Shojania receives general salary support from the Government of Canada Research Chairs Program. Neither funding body played any role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the systematic review.
Other disclosures:
None.
Ethical approval:
Not applicable.
Disclaimer:
The opinions expressed in this article are those of the authors alone and do not reflect the views of the Association of the Faculties of Medicine of Canada.
Previous presentations:
A preliminary version of the review appears as a chapter in a monograph entitled “The Future of Medical Education in Canada” (http://www.afmc.ca/fmec/activities-env-literature.php), published by the Association of Faculties of Medicine of Canada.
References
1Fletcher KE, Underwood W 3rd, Davis SQ, Mangrulkar RS, McMahon LFJ, Saint S. Effects of work hour reduction on residents' lives: A systematic review. JAMA. 2005;294:1088–1100.
2Ogden PE, Sibbitt S, Howell M, et al. Complying with ACGME resident duty hours restrictions: Restructuring the 80-hour workweek to enhance education and patient safety at Texas A&M/Scott & White Memorial Hospital. Acad Med. 2006;81:1026–1031.
3Okie S. An elusive balance—Residents' work hours and the continuity of care. N Engl J Med. 2007;356:2665–2667.
4Shojania KG, Fletcher KE, Saint S. Graduate medical education and patient safety: A busy—and occasionally hazardous—intersection. Ann Intern Med. 2006;145:592–598.
5Kennedy TJ, Regehr G, Baker GR, Lingard L. Preserving professional credibility: Grounded theory study of medical trainees' requests for clinical support. BMJ. 2009;338:b128.
6Vidyarthi AR, Arora V, Schnipper JL, Wall SD, Wachter RM. Managing discontinuity in academic medical centers: Strategies for a safe and effective resident sign-out. J Hosp Med. 2006;1:257–266.
7Association of American Medical Colleges. Report X: Contemporary Issues in Medicine: Education in Safe and Effective Prescribing Practices. Medical School Objectives Project. Available at:
http://www.aamc.org/meded/msop. Accessed March 24, 2010.
8Association of American Medical Colleges. Report V: Contemporary Issues in Medicine: Quality of Care. Medical School Objectives Project. Available at:
http://www.aamc.org/meded/msop. Accessed March 24, 2010.
9Accreditation Council for Graduate Medical Education. ACGME Outcome Project. Common Program Requirements: General Competencies. Available at:
http://www.acgme.org/outcome/comp/compCPRL.asp. Accessed March 24, 2010.
10Frank JR, Danoff D. The CanMEDS initiative: Implementing an outcomes-based framework of physician competencies. Med Teach. 2007;29:642–647.
11Scheele F, Teunissen P, Van Luijk S, et al. Introducing competency-based postgraduate medical education in the Netherlands. Med Teach. 2008;30:248–253.
12Shojania KG, Levinson W. Clinicians in quality improvement: A new career pathway in academic medicine. JAMA. 2009;301:766–768.
13Boonyasai RT, Windish DM, Chakraborti C, Feldman LS, Rubin HR, Bass EB. Effectiveness of teaching quality improvement to clinicians: A systematic review. JAMA. 2007;298:1023–1037.
14Kohn LT, Corrigan JM, Donaldson MS, eds. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press; 2000.
15Best Evidence in Medical Education (BEME) Collaboration: Starting a Review. Available at:
http://www2.warwick.ac.uk/fac/med/beme/startingreview/. Accessed May 10, 2010.
16Kirkpatrick DL. Evaluation of training. In: Craig RL, Bittel LR, eds. Training and Development Handbook. New York, NY: McGraw-Hill; 1967:87–112.
17Hughes MT. Implementation. In: Kern DE, Thomas PA, Howard DM, Bass EB, eds. Curriculum Development for Medical Education: A Six-Step Approach. Baltimore, Md: The Johns Hopkins University Press; 1998.
18Bechtold ML, Scott S, Nelson K, Cox KR, Dellsperger KC, Hall LW. Educational quality improvement report: Outcomes from a revised morbidity and mortality format that emphasised patient safety. Qual Saf Health Care. 2007;16:422–427.
19Canal DF, Torbeck L, Djuricich AM. Practice-based learning and improvement: A curriculum in continuous quality improvement for surgery residents. Arch Surg. 2007;142:479–482.
20Coleman MT, Nasraty S, Ostapchuk M, Wheeler S, Looney S, Rhodes S. Introducing practice-based learning and improvement ACGME core competencies into a family medicine residency curriculum. Jt Comm J Qual Saf. 2003;29:238–247.
21Coyle YM, Mercer SQ, Murphy-Cullen CL, Schneider GW, Hynan LS. Effectiveness of a graduate medical education program for improving medical event reporting attitude and behavior. Qual Saf Health Care. 2005;14:383–388.
22Djuricich AM, Ciccarelli M, Swigonski NL. A continuous quality improvement curriculum for residents: Addressing core competency, improving systems. Acad Med. 2004;79(10 suppl):S65–S67.
23Frey K, Edwards F, Altman K, Spahr N, Gorman RS. The ‘Collaborative Care’ curriculum: An educational model addressing key ACGME core competencies in primary care residency training. Med Educ. 2003;37:786–789.
24Gould BE, Grey MR, Huntington CG, et al. Improving patient care outcomes by teaching quality improvement to medical students in community-based practices. Acad Med. 2002;77:1011–1018.
25Gunderson A, Tekian A, Mayer D. Teaching interprofessional health science students medical error disclosure. Med Educ. 2008;42:531–532.
26Halbach JL, Sullivan LL. Teaching medical students about medical errors and patient safety: Evaluation of a required curriculum. Acad Med. 2005;80:600–606.
27Henley E. A quality improvement curriculum for medical students. Jt Comm J Qual Improv. 2002;28:42–48.
28Holmboe ES, Prince L, Green M. Teaching and improving quality of care in a primary care internal medicine residency clinic. Acad Med. 2005;80:571–577.
29Kerfoot BP, Conlin PR, Travison T, McMahon GT. Web-based education in systems-based practice: A randomized trial. Arch Intern Med. 2007;167:361–366.
30Madigosky WS, Headrick LA, Nelson K, Cox KR, Anderson T. Changing and sustaining medical students' knowledge, skills, and attitudes about patient safety and medical fallibility. Acad Med. 2006;81:94–101.
31Mohr JJ, Randolph GD, Laughon MM, Schaff E. Integrating improvement competencies into residency education: A pilot project from a pediatric continuity clinic. Ambul Pediatr. 2003;3:131–136.
32Moskowitz E, Veloski JJ, Fields SK, Nash DB. Development and evaluation of a 1-day interclerkship program for medical students on medical errors and patient safety. Am J Med Qual. 2007;22:13–17.
33Newell P, Harris S, Aufses AJ, Ellozy S. Student perceptions of medical errors: Incorporating an explicit professionalism curriculum in the third-year surgery clerkship. J Surg Educ. 2008;65:117–119.
34Ogrinc G, Headrick LA, Morrison LJ, Foster T. Teaching and assessing resident competence in practice-based learning and improvement. J Gen Intern Med. 2004;19:496–500.
35Ogrinc G, West A, Eliassen MS, Liuw S, Schiffman J, Cochran N. Integrating practice-based learning and improvement into medical student learning: Evaluating complex curricular innovations. Teach Learn Med. 2007;19:221–229.
36Oyler J, Vinci L, Arora V, Johnson J. Teaching internal medicine residents quality improvement techniques using the ABIM's practice improvement modules. J Gen Intern Med. 2008;23:927–930.
37Patey R, Flin R, Cuthbertson BH, et al. Patient safety: Helping medical students understand error in healthcare. Qual Saf Health Care. 2007;16:256–259.
38Peters AS, Kimura J, Ladden MD, March E, Moore GT. A self-instructional model to teach systems-based practice and practice-based learning and improvement. J Gen Intern Med. 2008;23:931–936.
39Tomolo A, Caron A, Perz ML, Fultz T, Aron DC. The outcomes card. Development of a systems-based practice educational tool. J Gen Intern Med. 2005;20:769–771.
40Varkey P, Karlapudi SP, Bennet KE. Teaching quality improvement: A collaboration project between medicine and engineering. Am J Med Qual. 2008;23:296–301.
41Varkey P, Reller MK, Smith A, Ponto J, Osborn M. An experiential interdisciplinary quality improvement education initiative. Am J Med Qual. 2006;21:317–322.
42Voss JD, May NB, Schorling JB, et al. Changing conversations: Teaching safety and quality in residency training. Acad Med. 2008;83:1080–1087.
43Weingart SN, Tess A, Driver J, Aronson MD, Sands K. Creating a quality improvement elective for medical house officers. J Gen Intern Med. 2004;19:861–867.
44Ziegelstein RC, Fiebach NH. “The mirror” and “the village”: A new method for teaching practice-based learning and improvement and systems-based practice. Acad Med. 2004;79:83–88.
45Cosby KS, Croskerry P. Patient safety: A curriculum for teaching patient safety in emergency medicine. Acad Emerg Med. 2003;10:69–78.
46Esselman PC, Dillman-Long J. Morbidity and management conference: An approach to quality improvement in brain injury rehabilitation. J Head Trauma Rehabil. 2002;17:257–262.
47Farquhar D, Myers K, Benjamin D. Education in quality of care in an internal medicine residency program. Acad Med. 2001;76:562.
48Gosbee J. A patient safety curriculum for residents and students: The VA healthcare system's pilot project. ACGME Bulletin. November 2002:2–6.
49Gould BE, O'Connell MT, Russell MT, Pipas CF, McCurdy FA. Teaching quality measurement and improvement, cost-effectiveness, and patient satisfaction in undergraduate medical education: The UME-21 experience. Fam Med. 2004;36(suppl):S57–S62.
50Krajewski K, Siewert B, Yam S, Kressel HY, Kruskal JB. A quality assurance elective for radiology residents. Acad Radiol. 2007;14:239–245.
51Paulman P, Medder J. Teaching the quality improvement process to junior medical students: The Nebraska experience. Fam Med. 2002;34:421–422.
52Rosenfeld JC. Using the Morbidity and Mortality conference to teach and assess the ACGME General Competencies. Curr Surg. 2005;62:664–669.
53Schillinger D, Wheeler M, Fernandez A. The populations and quality improvement seminar for medical residents. Acad Med. 2000;75:562–563.
54Singh R, Naughton B, Taylor JS, et al. A comprehensive collaborative patient safety residency curriculum to address the ACGME core competencies. Med Educ. 2005;39:1195–1204.
55Thompson DA, Cowan J, Holzmueller C, Wu AW, Bass E, Pronovost P. Planning and implementing a systems-based patient safety curriculum in medical education. Am J Med Qual. 2008;23:271–278.
56Varkey P. Educating to improve patient care: Integrating quality improvement into a medical school curriculum. Am J Med Qual. 2007;22:112–116.
57Weeks WB, Robinson JL, Brooks WB, Batalden PB. Using early clinical experiences to integrate quality-improvement learning into medical education. Acad Med. 2000;75:81–84.
58Wong RY, Hollohan K, Roberts M, Hatala R, Ma IW, Kassen BO. A descriptive report of an innovative curriculum to teach quality improvement competencies to internal medicine residents. Can J Gen Intern Med. 2008;3:26–29.
59Morrison LJ, Headrick LA, Ogrinc G, Foster T. The quality improvement knowledge application tool: An instrument to assess knowledge application in practice-based learning and improvement. J Gen Intern Med. 2003;18(suppl 1):250.
60Marinopoulos SS, Dorman T, Ratanawongsa N, et al. Effectiveness of Continuing Medical Education. Available at:
http://www.ahrq.gov/clinic/tp/cmetp.htm. Accessed March 24, 2010.
61Belfield C, Thomas H, Bullock A, Eynon R, Wall D. Measuring effectiveness for best evidence medical education: A discussion. Med Teach. 2001;23:164–170.
62Davis D. Does CME work? An analysis of the effect of educational activities on physician performance or health care outcomes. Int J Psychiatry Med. 1998;28:21–39.
63Davis D, O'Brien MA, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. Impact of formal continuing medical education: Do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA. 1999;282:867–874.
64Schouten LM, Hulscher ME, van Everdingen JJ, Huijsman R, Grol RP. Evidence for the impact of quality improvement collaboratives: Systematic review. BMJ. 2008;336:1491–1494.
65Wu AW, Lipshutz AK, Pronovost PJ. Effectiveness and efficiency of root cause analysis in medicine. JAMA. 2008;299:685–687.
66Brainard AH, Brislen HC. Viewpoint: Learning professionalism: A view from the trenches. Acad Med. 2007;82:1010–1014.
67D'Eon M, Lear N, Turner M, Jones C. Perils of the hidden curriculum revisited. Med Teach. 2007;29:295–296.
68Gofton W, Regehr G. What we don't know we are teaching: Unveiling the hidden curriculum. Clin Orthop Relat Res. 2006;449:20–27.
69Nazem AG. A piece of my mind. Teach us how. JAMA. 2008;300:2463–2464.
70Alper E, Rosenberg EI, O'Brien KE, Fischer M, Durning SJ. Patient safety education at U.S. and Canadian medical schools: Results from the 2006 Clerkship Directors in Internal Medicine survey. Acad Med. 2009;84:1672–1676.
