Purpose: Incorporating quality improvement (QI) into resident education and clinical care is challenging. This report explores key characteristics shaping the relative success or failure of QI efforts in seven primary care practices serving as family medicine residency training sites.
Method: The authors used data from the 2002–2008 Using Learning Teams for Reflective Adaptation study to conduct a comparative case analysis. This secondary data analysis focused on seven residency training practices' experiences with the reflective adaptive process (RAP), a 12-week intensive QI process. Field notes, meeting notes, and audiotapes of RAP meetings were used to construct case summaries. A matrix comparing key themes across practices was used to rate practices' QI progress during RAP on a scale of 0 to 3.
Results: Three practices emerged as unsuccessful (scores of 0–1) and four as successful (scores of 2–3). Larger practices with previous QI experience, faculty with extensive exposure to QI literature, and an office manager, residency director, or medical director who advocated the process made substantial progress during RAP, succeeding at QI. Smaller practices without these characteristics were unable to do so. Successful practices also engaged residents in the QI process and identified serious problems as potential crises; unsuccessful practices did not.
Conclusions: Larger residency training practices are more likely to have the resources and characteristics that permit them to create a QI-supportive culture leading to QI success. The authors suggest, however, that smaller practices may increase their chances of success by adopting a developmental approach to QI.
Dr. Chase is medical anthropologist and research analyst, Research Division, Department of Family Medicine, University of Medicine and Dentistry of New Jersey, Robert Wood Johnson Medical School, New Brunswick, New Jersey.
Dr. Miller is Leonard Parker Pool Chair of Family Medicine, Lehigh Valley Health Network, Allentown, Pennsylvania, and professor of family medicine, University of South Florida College of Medicine, Tampa, Florida.
Dr. Shaw is assistant professor, Research Division, Department of Family Medicine, University of Medicine and Dentistry of New Jersey, Robert Wood Johnson Medical School, New Brunswick, New Jersey.
Dr. Looney is assistant professor, Predoctoral Education Division, Department of Family Medicine, University of Medicine and Dentistry of New Jersey, Robert Wood Johnson Medical School, Piscataway, New Jersey.
Dr. Crabtree is professor of family medicine, University of Medicine and Dentistry of New Jersey, Robert Wood Johnson Medical School, professor of epidemiology, University of Medicine and Dentistry of New Jersey School of Public Health, and cancer prevention and control program leader, Cancer Institute of New Jersey, New Brunswick, New Jersey.
Please see the end of this article for information about the authors.
Correspondence should be addressed to Dr. Chase, Robert Wood Johnson Medical School, Department of Family Medicine, Research Division, 1 Worlds Fair Dr., Somerset, NJ 08873; telephone: (732) 309-5032; fax: (732) 743-3395; e-mail: firstname.lastname@example.org.
First published online October 25, 2011
Academic family medicine residency programs must operate under an array of burdens. They struggle with dwindling resources,1,2 lower-than-optimal interest from medical students,3,4 and the competing demands of education and patient care. At the same time, they must attempt to stay abreast of changing guidelines,5 innovative curricula,6,7 and new developments such as the emergence of patient-centered medical homes (PCMHs).8–12 Meanwhile, the Accreditation Council for Graduate Medical Education and the Alliance of Independent Academic Medical Centers continue to emphasize the importance of weaving quality improvement (QI) initiatives into resident education and clinical care.13,14 To implement successful QI in residency training15 while managing all these competing demands, residency programs and their associated training sites are increasingly looking for new ways to rise to the challenge of change.16–21
Organizational change is not a linear process. Conventional approaches to change often treat organizations like machines with broken parts that need to be repaired,22 but this ignores their complexity23–25 and the challenges inherent in sustaining new efforts over time.26 Recently, organizational change at the practice level has been identified as a developmental process, as noted in work surrounding the American Academy of Family Physicians' first National Demonstration Project (NDP), which focused on generating new models of the PCMH.12 NDP evaluators found that even highly motivated, resource-rich physician practices needed to take small steps toward change before attempting large-scale shifts.11,27
A recent multiyear study known as Using Learning Teams for Reflective Adaptation (ULTRA) targeted QI at the practice level by drawing on common QI methods and the insights of complexity science to lead diverse, primary care practice-based teams through a 12-week action/reflection cycle.22,28 Here, we examine key details of the QI implementation process in those ULTRA practices that were also family medicine residency training sites in order to identify practice characteristics that acted as barriers to and facilitators of QI success. Our findings add to an emerging body of literature illuminating how QI methods are chosen and why change efforts succeed or fail.16–21 They also have important implications for residency training practices seeking to adopt QI initiatives of their own while maintaining their many day-to-day responsibilities.
ULTRA was a group-randomized, delayed-intervention QI trial that took place during 2002–2008; full details of the study have been published previously.29,30 Sixty-two primary care practices (31 intervention sites, 31 controls) were recruited into the study. All were located in New Jersey and Pennsylvania and affiliated with the New Jersey Family Medicine Research Network or the Lehigh Valley Health Network. The seven practices that are the focus of this article were family medicine residency training sites.
ULTRA's QI intervention, which was initially introduced into the intervention practices and later into the control practices, incorporated a multimethod assessment process (MAP)31 and facilitated QI teams known as reflective adaptive process (RAP) teams.22 Thus, the seven family medicine residency training practices described here all received the MAP/RAP intervention. The University of Medicine and Dentistry of New Jersey and Lehigh Valley Health Network institutional review boards reviewed and approved data collection and analysis protocols for the ULTRA study.
MAP/RAP began with MAP, which involved the collection of a wide range of qualitative data by an ULTRA facilitator, including field notes based on two weeks of observation in the practice, audiotaped formal interviews of key practice members, and notes on informal interviews. To ensure comparability across all ULTRA practices, facilitators used an observation template that prompted them to observe office organization and procedures, ask about decision making in the practice, and describe the physical setting of the office. Throughout MAP and, later, RAP, facilitators attended weekly meetings of the larger ULTRA study team to exchange findings and identify areas for follow-up data collection.
Facilitators then worked with the larger research team to generate a concise MAP summary report (MSR) for each practice. This report summarized the practice's values, strategic vision, readiness to change, motivators, relationships, and procedures. The facilitator shared and discussed the MSR with practice members immediately after the completion of the MAP cycle and before the initiation of RAP. Because ULTRA was a delayed-intervention study, MSRs were generated for both intervention and control practices.
For RAP, each practice assembled a 7- to 11-member team that included representatives from all parts of the practice as well as the patient community. During hour-long weekly meetings held over the course of about three months, an ULTRA facilitator guided each RAP team through the application of practical QI tools and methods to address persistent practice-level problems identified by the team.28,32 The facilitated teams used plan–do–check–act cycles informed by MAP findings to create targeted action plans and generate strategies for evaluating and modifying these plans until success was attained. A critical goal of RAP was to provide space and time for team members to engage in ongoing learning conversations that bolstered the practice's “adaptive reserve,” or ability to respond effectively to turbulence and rapid change.27 These regular meetings encouraged the development of facilitative leadership (a leadership style that emphasizes open communication, collaboration, and teamwork),33 a learning culture,34 and the ability to improvise. RAP differs from other QI initiatives in its intense focus on each of these domains and in its use of external facilitation. All RAP team meetings were audiotaped, and the facilitator described each meeting in a set of written field notes.
At six months, one year, and two years after the completion of RAP, facilitators followed up with each practice to conduct interviews and assess longer-term effects of the intervention. Facilitators asked how RAP had affected work relationships and practice-wide communication, probing for RAP's effect on the practice as a whole.
This report presents a secondary analysis of the original ULTRA study data: We conducted a comparative case analysis that focused only on the seven family medicine residency training practices that participated in the larger ULTRA study. Data for this secondary analysis included written MAP field notes, written RAP meeting field notes, and audiotapes of RAP meetings. Our research team used an editing style of analysis,35,36 reading a selection of all available field notes for the seven practices. One researcher (S.M.C.) read all field notes and constructed case summaries for each of the seven practices. In addition, she listened to a selection of audiotaped RAP meetings when more detail was needed to evaluate a team's progress in establishing and meeting significant QI change targets. Our team met regularly to review the case summaries and identify emerging themes. When all seven case summaries were completed, we met again to review the data and rate each practice's QI progress during RAP, which we defined as its ability to conduct consistent, widely representative RAP team meetings; identify areas requiring improvement; translate these areas into specific change targets; meet or exceed established change targets; and incorporate resulting improvements into the practice as a whole. Analytical categories emerged as we examined the data, and a substantial gap quickly appeared between those practices that were able to accomplish all or most of the above and those that were not. To facilitate our analysis, we constructed a matrix comparing ratings and key themes across all seven training practices.37
We rated each practice using a scale of 0 to 3: Practices that were unable to accomplish the above items and that implemented no improvements (0) or adopted only a few insignificant changes (1) were designated as unsuccessful, or “QI failures.” Practices whose RAP teams met consistently and identified and implemented a complete series of moderate (2) or substantial (3) practice improvements were designated as successful, or “QI successes.” No middle category emerged.
Table 1 presents the seven practice characteristics that emerged from our analysis of the seven case summaries. It also provides our ratings of each residency training practice's QI progress during RAP (i.e., its ability to implement RAP successfully).
As Table 1 indicates, three practices received scores of 0 or 1 (unsuccessful/QI failure), and four practices received a 2 or 3 (successful/QI success). Among the three practices designated as QI failures, one practice did not complete the intervention, and another did not meet its central change target after more than a year of RAP meetings. The third practice completed the intervention and established very minor change targets, but it failed to implement significant improvements.
All four of the practices designated as QI successes completed the intervention, established multiple, significant change targets, and implemented them throughout the practice. Additionally, three held effective RAP meetings for well over a year (practices 4, 6, 7; data not shown). Three of the successful practices received a score of 3. The fourth received a 2 because our ULTRA facilitator's follow-up investigation revealed that the benefits of RAP extended unevenly across the organization.
As Table 1 shows, there was a clear split between those residency training practices that succeeded at RAP and those that failed. These differences are highlighted in Case Study 1 and Case Study 2, which respectively provide examples of one unsuccessful and one successful practice improvement cycle.
The four large residency training practices (with 18–24 residents) made substantial progress during RAP. They all had previous QI team experience and/or faculty with extensive exposure to QI literature. Three had consistent resident participation in RAP meetings. In addition, three publicly recognized and grappled with a practice-wide crisis during RAP. All four successful practices had at least one “RAP champion”—an office manager, residency director, or family medicine department chair—who believed in the utility of RAP and encouraged forward movement. In this study, we defined a RAP champion as a highly active advocate of QI-related activities—specifically, someone who ensured that RAP team meetings consistently took place as scheduled, that team members remained actively involved, and that action plans were executed as designed. A RAP champion's commitment was often infectious, and, in some cases, his or her belief in the QI process helped sustain the entire RAP team as it realized that the practice was in crisis (for an example, see Case Study 2). Two successful practices had secondary supporters as well (their medical directors).
The three small residency training practices (with 7–9 residents) struggled to hold consistent RAP team meetings and/or were unable to translate areas requiring improvement into achievable change targets. Each failed to make substantial improvements across the practice as a whole and essentially made little or no QI progress during RAP. None of these practices had a RAP champion. At one practice, the medical director and office manager regularly shut down RAP team discussions by being openly critical and judgmental of staff. For example, the medical director stymied an early RAP session by declaring, “Some people are just hares and some are tortoises. Tortoises are never [going to] get any better because they don't want to change.” As a result of this communication pattern, team members stopped participating, and team conversations remained guarded. The teams at the other two practices also struggled to collaborate. For example, facilitator field notes from one of these practices recorded the following conversation with a RAP team nurse:
She [the nurse] told me that the time the team spent working on reducing lost charts had no effect at all. “Nobody's willing to change anything,” [the nurse] explained. She told me that the RAP team tried to have charts filed in one area so there would be fewer places in which to look for them. “[But],” she pointed out, “We've moved them half a dozen times'cause somebody thinks they should be here, some there ... people will say, where is the ... chart box this week?”
Among the three unsuccessful practices, only one had previous experience instituting a QI team, and none showed evidence of faculty with extensive exposure to QI literature. None featured ongoing, active resident participation on the RAP team; in one case, the medical director actually excluded residents from the team, despite at least one resident's declaration of interest. In addition, although individuals at the unsuccessful practices often discussed serious, ongoing problems during RAP team meetings, neither their practice leadership nor their RAP teams publicly identified these problems as practice-wide crises worthy of a sustained, collaborative response. This meant that deep, recurring problems went unresolved. For example, one practice was plagued by a deep, disruptive distrust. During one RAP meeting, the nurse supervisor proposed that a time stamp be used to record patient arrivals because the nurses suspected that the front desk staff were lying about patient arrival times. The front desk supervisor responded that it was ridiculous to order another stamp—The problem was that the nurses needed to trust the front desk staff. More than half the meeting was spent discussing lack of trust among practice members. At the meeting's close, the team decided to form a subcommittee to clarify what was happening, but no concrete plan was ever offered to facilitate this or to address the distrust. Finally, all three unsuccessful practices espoused a “nothing will ever change” philosophy. Practice members in each of these organizations repeatedly shrugged off the suggestion that collaboration and communication could lead to practice improvements. It should be noted, however, that one of the QI successes shared this attitude.
To succeed at RAP, the family medicine residency training practices in our sample were required to shift a pool of additional resources (e.g., working hours, personnel focus, QI experience) into their QI efforts while continuing to teach and supervise residents. They had to learn to recognize urgent practice problems as actual or potential crises and use the RAP process to address them. When they faced financial and staffing constraints, as in Case Study 2, the challenges they confronted seemed greatly magnified, and the QI process seemed even more daunting. As our analysis shows, the larger residency training practices were better able to meet these challenges and enjoyed greater overall success in their QI efforts than their smaller counterparts. But this does not mean that smaller residency training practices cannot succeed at QI.
Careful consideration of the NDP practice development model27 can shed additional light on how residency training practices can undertake successful, long-term practice improvement efforts. In the NDP study of 36 family medicine practices, evaluators found that accomplishing this required that both leaders and practice members attend to a wide range of factors, including a practice's core characteristics (its resources, organizational components, and functional processes), its “adaptive reserve” (those relationships and features that enhance resilience),27,38 and its local environment (hospital systems, area employers, public events, and community agencies).8,39–41 The NDP practice development model proposes that primary care practices are complex adaptive systems22 that can be broken down into these three components. Although all working practices must have a functioning core, the strength of an organization's adaptive reserve and its attention to its local environment can vary widely. In the NDP study, practices with strong adaptive reserves that were already well connected to their local environments were able to make the most far-reaching changes.11
We believe that the NDP findings hold true for ULTRA's family medicine residency training practices as well. Our analysis showed that those practices that held regular meetings, engaged residents, and maximized their faculty's exposure to QI literature and practice while juggling dual missions of patient care and resident education were more successful at RAP than the others. In our study, these successful practices were embedded in larger residency programs partnered with more extensive, resource-rich hospital systems. These practices may have been better able to distribute QI planning work among their more numerous faculty members. Large practices also seemed to experience fewer barriers when attempting to involve residents in practice QI efforts than did small practices. All of the practices identified problems, but the large practices were more likely to publicly recognize that their organizations faced serious crises requiring sustained, collaborative responses than were their smaller counterparts. This suggests that, just as in the NDP study, larger practices with stronger adaptive reserves had more resources to bestow on QI efforts. Because their adaptive reserves were initially stronger, the larger practices could devote the time, energy, and focus demanded by the RAP process and thereby reap the greater benefits of this particular QI approach. Smaller practices with fewer resources seemed to be struggling simply to maintain their practice cores and, thus, were unable to muster the resources needed to succeed at RAP.
However, smaller residency training practices can, we believe, increase their chances of success if they take note of the NDP's findings by adopting a developmental perspective regarding organizational change. Instead of beginning with a resource-intensive style of QI such as RAP, we suggest that they start with a more basic approach. Visual management strategies, such as those used by Toyota Lean,42,43 might be an appropriate initial investment. These techniques would allow practices to choose a small number of change targets and monitor them daily through the use of “information radiators”44 (e.g., tracking boards), together with short, frequent meetings. Because visual management does not require an influx of new resources45 and is easily used as a day-to-day management tool,44,46 it is a particularly useful approach for smaller practices. Basic approaches like these will also allow smaller residency training practices to familiarize faculty and other practice members with the QI literature, acquire practical experience, bolster their cores, and begin to explore a facilitative style of leadership—all of which develop adaptive reserve. As such practices become more proficient in their QI efforts, they can adopt more labor-intensive styles of QI.
Our study also highlights the important role of leadership in sustaining the QI process. Each successful residency training practice had at least one highly motivated leader–advocate (the “RAP champion”). Office managers, medical directors, and residency directors played critical roles in sustaining—or undermining—QI efforts. In this sense, practice development sat largely in their hands. As RAP champions, they made sure that meetings happened: They urged discussion forward at key moments and pulled residents, faculty, and staff into the process. Their commitment was infectious, and, in some cases, their belief in the process helped revive practices in crisis (see Case Study 2 for an example). In stark contrast, medical directors and office managers who blocked open discussion or offered little enthusiasm strangled the RAP process before it could produce results.
Finally, our analysis suggests that active resident participation is important to the success of QI at residency training practices. Those practices that integrated residents into their RAP teams on an ongoing basis produced generally better results than did those that either excluded them or did not actively encourage their participation.
Limitations of this study include its small sample size and the constraints of secondary analysis. We should note that these seven primary care practices may not be representative of all U.S. residency training practices: They were a subsample of the larger ULTRA study, which was not originally designed to evaluate the success of QI in residency training practices.
Nevertheless, our study demonstrates that QI initiatives in family medicine residency training settings can be successful despite the challenges they face. However, each residency training practice must take care to adopt the strategy most likely to be successful in its own particular context. If residency training practices adopt a developmental approach as NDP findings suggest, they may be able to use QI to strengthen their cores and build their adaptive reserves over time. Further, encouraging residents to participate in QI initiatives offers them a practical set of skills that can prepare them to practice in our continually shifting health care landscape.
The ULTRA study on which this secondary analysis is based was funded by the National Heart, Lung, and Blood Institute (R01 HL70800). Dr. Crabtree's time was supported in part by a K05 grant from the National Cancer Institute (K05 CA140237).
The University of Medicine and Dentistry of New Jersey and Lehigh Valley Health Network institutional review boards reviewed and approved data collection and analysis protocols for the ULTRA study.
1 Pugno PA, Epperly TD. Residency Review Committee for Family Medicine: An analysis of program citations. Fam Med. 2005;37:174–177.
3 Bodenheimer T. Primary care—Will it survive? N Engl J Med. 2006;355:861–864.
4 Woo B. Primary care—The best job in medicine? N Engl J Med. 2006;355:864–866.
5 Østbye T, Yarnall KS, Krause KM, Pollak KI, Gradison M, Michener JL. Is there time for management of patients with chronic diseases in primary care? Ann Fam Med. 2005;3:209–214.
6 Carney PA, Eiff MP, Saultz JW, et al. Aspects of the patient-centered medical home currently in place: Initial findings from preparing the personal physician for practice. Fam Med. 2009;41:632–639.
7 Whitcomb ME. Preparing the personal physician for practice (P4): Meeting the needs of patients: Redesign of residency training in family medicine. J Am Board Fam Med. 2007;20:356–364.
8 Pham HH. Good neighbors: How will the patient-centered medical home relate to the rest of the health-care delivery system? J Gen Intern Med. 2010;25:630–634.
9 Dentzer S. Reinventing primary care: A task that is far “too important to fail.” Health Aff (Millwood). 2010;29:757.
10 Jackson JL. The patient-centered medical home and our future health-care system. J Gen Intern Med. 2010;25:483.
11 Crabtree BF, Nutting PA, Miller WL, Stange KC, Stewart EE, Jaen CR. Summary of the National Demonstration Project and recommendations for the patient-centered medical home. Ann Fam Med. 2010;8(suppl 1):S80–S90.
12 Stange KC, Miller WL, Nutting P, Crabtree BF, Stewart EE, Jaén CR. Context for understanding the National Demonstration Project and the patient-centered medical home. Ann Fam Med. 2010;8(suppl 1):S2–S7.
17 Tomolo AM, Lawrence RH, Aron DC. A case study of translating ACGME practice-based learning and improvement requirements into reality: Systems quality improvement projects as the key component to a comprehensive curriculum. Postgrad Med J. 2009;85:530–537.
20 Shunk R, Dulay M, Julian K, et al. Using the American Board of Internal Medicine practice improvement modules to teach internal medicine resident practice improvement. J Grad Med Educ. 2010;2:90–95.
21 Oyler J, Vinci L, Johnson JK, Arora VM. Teaching internal medicine residents to sustain their improvement through the quality assessment and improvement curriculum. J Gen Intern Med. 2010;26:221–225.
22 Stroebel CK, McDaniel RR Jr, Crabtree BF, Miller WL, Nutting P, Stange KC. How complexity science can inform a reflective process for improvement in primary care practices. Jt Comm J Qual Patient Saf. 2005;31:438–446.
23 Glasgow RE, Lichtenstein E, Marcus AC. Why don't we see more translation of health promotion research into practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health. 2003;93:1261–1267.
24 McCormack B, Kitson A, Harvey G, Rycroft-Malone J, Titchen A, Seers K. Getting evidence into practice: The meaning of “context.” J Adv Nurs. 2002;38:94–104.
25 Solberg LI, Brekke ML, Fazio CJ, et al. Lessons from experienced guideline implementers: Attend to many factors and use multiple strategies. Jt Comm J Qual Improv. 2000;26:171–188.
26 Doyle M, Claydon T, Buchanan D. Mixed results, lousy process: The management experience of organizational change. Br J Manage. 2000;11(suppl 1):S59–S80.
27 Miller WL, Crabtree BF, Stange KC, Nutting PA, Jaen CR. Primary care practice development: A relationship-centered approach. Ann Fam Med. 2010;8(suppl 1):S68–S79.
28 Chase SM, Nutting PA, Crabtree BF. How to solve problems in your practice with a new meeting approach. Fam Pract Manage. 2010;17:31–34.
29 Balasubramanian BA, Chase SM, Nutting PA, et al. Using learning teams for reflective adaptation (ULTRA): Insights from a team-based change management strategy in health care. Ann Fam Med. 2010;8:425–432.
30 Shaw E, Looney A, Chase S, Navalekar R, Stello B, Crabtree B. “In the moment”: An analysis of facilitator impact during a quality improvement process. Group Facil. 2010;10:4–16.
31 Crabtree BF, Miller WL, Stange KC. Understanding practice from the ground up. J Fam Pract. 2001;50:881–887.
32 Scholtes P, Joiner BL, Streiber BJ. The Team Handbook. 3rd ed. Madison, Wis: Oriel, Inc.; 2003.
33 Wilson PH. The Facilitative Way: Leadership That Makes a Difference. Shawnee Mission, Kan: Team Tech, Inc.; 2003.
35 Addison RB. A grounded hermeneutic editing approach. In: Crabtree BF, Miller WL, eds. Doing Qualitative Research. 2nd ed. Thousand Oaks, Calif: Sage; 1999:145–161.
36 Miller WL, Crabtree BF. Clinical research: A multimethod typology and qualitative roadmap. In: Crabtree BF, Miller WL, eds. Doing Qualitative Research. 2nd ed. Thousand Oaks, Calif: Sage; 1999.
37 Miles MB, Huberman AM. Qualitative Data Analysis: An Expanded Sourcebook. Thousand Oaks, Calif: Sage; 1994.
38 Lanham HJ, McDaniel RR Jr, Crabtree BF, et al. How improving practice relationships among clinicians and nonclinicians can improve quality in primary care. Jt Comm J Qual Patient Saf. 2009;35:457–466.
39 Cohen D, McDaniel RR Jr, Crabtree BF, et al. A practice change model for quality improvement in primary care practice. J Healthc Manag. 2004;49:155–168.
40 Fisher ES. Building a medical neighborhood for the medical home. N Engl J Med. 2008;359:1202–1205.
41 Crabtree BF, Miller WL, McDaniel RR, Stange KC, Nutting PA, Jaen CR. A survivor's guide for primary care physicians. J Fam Pract. August 2009;58:E1.
42 Chalice R. Improving Healthcare Using Toyota Lean Production Methods: 46 Steps for Improvement. 2nd ed. Milwaukee, Wis: American Society for Quality Press; 2007.
43 Liff S, Posey P. Seeing Is Believing: How the New Art of Visual Management Can Boost Performance Throughout Your Organization. New York, NY: AMACOM; 2004.
© 2011 Association of American Medical Colleges
46 Packman Z. Seeing is believing. Nurs Manag (Harrow). 2007;14:20–23.