Asao, Keiko MD, MPH, PhD; Mansi, Ishak A. MD; Banks, Daniel MD, MS
The Accreditation Council for Graduate Medical Education1 and several other organizations2,3 have invited quality improvement (QI) initiatives in residency programs and clinical practices. However, knowledge about how to initiate and sustain QI in residency programs4 is limited. Efforts to incorporate QI initiatives in residency training have taken many forms—inclusion of residents on hospital QI committees,5 residency program improvement driven by the residents themselves,6 medical record audit with7,8 or without9,10 extensive and formal didactic sessions, and use of Web-based tools and databases,11 to name a few. In addition, and partially as a consequence of the absence of an established intervention format, few reports have demonstrated the clinical and educational effectiveness of QI initiatives in residency programs.12,13
Nevertheless, the need for understanding and implementing QI at all levels is imperative. In the 2001 report, Crossing the Quality Chasm: A New Health System for the 21st Century,2 the Institute of Medicine delineated the framework of QI with six dimensions: safe, effective, patient-centered, timely, efficient, and equitable. QI initiatives that followed were brought on by rapid advancement of medical science and technology, limited available resources for health care, and the shift in management from acute to chronic diseases.2 Many efforts have been made to encourage the expansion of QI efforts. The Agency for Healthcare Research and Quality14 and many professional organizations have set QI indices. A survey showed that 52% of commercial health maintenance organizations (HMOs) have implemented pay for performance, a system that pays providers for carrying out high-quality care.15 The Centers for Medicare and Medicaid Services initiated a physician quality reporting system with a financial incentive in 2007.16 These initiatives are important steps toward improving quality, but high-quality practices must be cultivated early on in every health care provider's training. It is likely that graduate medical education has a significant impact on a physicians' lifelong performance in quality of care, as previous studies showed an association among residency program directors' ratings of clinical competencies, board certification examination scores, and quality-of-care indices after completion of graduate medical education.17–19
Therefore, the aim of our initiative was to implement a QI program within an internal medicine residency program through a medical record audit by peers, incorporating a simple design, minimal didactic teaching, and feedback with a motivational plan. Our goal in this report is to examine the effects of the QI intervention and to investigate the factors associated with QI indices.
The QI intervention comprised three parts: a didactic session, a medical record audit, and feedback. All residents in the internal medicine and the internal medicine–pediatrics programs of Louisiana State University Health Sciences Center in Shreveport (LSUHSC-S), an urban university hospital, were eligible to participate. We performed a medical record audit in the residents' continuity clinic, where all residents hold a half-day clinic each week for the entire period of training. Patients seen at the clinic are predominantly uninsured or covered by Medicare or Medicaid. The intervention was performed twice for two consecutive months at a time in 2007 (phase 1: throughout March and April; and phase 2: throughout August and September) using the same protocol, which is described below. Because we implemented the intervention during two academic years, residents who were inthe programs for those two years were involved in both phases, whereas the residents who left or entered after the first phase were involved only in one phase.
We held a one-hour didactic session before beginning each phase. At the session, a faculty member (I.M.) addressed QI principles, the scientific rationale of the selected QI indices, the audit and scoring process, and a motivational plan. These factors are described in detail below. We held the session as one of our regular educational lectures, and we encouraged every resident to attend. In addition, we distributed a copy of the project protocol to all residents by e-mail.
The institutional review board for human research subjects at LSUHSC-S approved this project.
We adopted 24 quality-of-care indices in the following five diagnoses: chronic obstructive pulmonary disease (COPD),20 coronary heart disease (CHD),20 diabetes mellitus (DM),20,21 hypertension (HTN),20 and left ventricular failure (LVF).20,22 We selected those diagnoses because they are common conditions at the residents' clinic and the indices for those diagnoses were well investigated. Diagnosis-specific indices included 3, 6, 8, 1, and 6 items, respectively (Table 1). Resident auditors answered either “yes” or “no” to indicate whether a specific quality index was documented in a chart. We defined documentation as any entry in the medical records within the last six months of the index visit.
During each two-month intervention period, a designated nurse pulled a convenience sample of one to three medical records for each participating resident to audit on each clinic day. We did not inform the nurse about the details of the project, including the indices monitored, the process, orthe results of the project. We excluded patients from the convenience sample if they were attending a posthospital discharge clinic or a clinic at which the usual resident providing care was absent. At the time of study, the clinic's medical records were partially electronic: Residents could retrieve laboratory data, discharge summaries, and the patient's clinic appointment schedule from a computerized system, but all other clinic records were paper based. On the basis of residents' availability, we assigned one resident to audit charts instead of performing the usual clinical duties each day. To be eligible, participants had to be second-year residents or above in the ambulatory care clinic rotation. Because this is the first attempt of this kind of intervention in our residency programs, most residents did not have previous experience auditing medical records.
Feedback and motivational plan
The project included opportunities for residents to receive feedback based on the results of the audits and a motivational plan to address consistently high or low compliance with QI indices. The chief resident and a faculty member (I.M.) reviewed all audit results. They provided ongoing feedback to each resident by a short phone call without any structural format. The feedback was given after reviewing multiple charts for an individual resident during the two-month period of the project. The feedback conversation could include residents' refutation of the review.
Initially, we intended to base the motivational plan around the schedule of clinical duties and vacations, and the choices in elective rotations for those who achieved the highest and lowest compliance scores with QI indices. However, after collecting the data, we decided not to implement the plan because of the small sample sizes of individual residents' work.
We expressed the total and diagnosis-specific scores as the proportion of the items that satisfied the standard quality of care, multiplied by 100. On the basis of the number of QI indices specific to each diagnosis, we applied a weight of 1/15, 1/30, 1/40, 1/5, and 1/30 for each item of COPD, CHD, DM, HTN, and LVF, respectively, so that each diagnosis equally contributed to the total score.
We compared the total and diagnosis-specific scores by month of clinic visit, prior experience of the resident as an auditor, duration of training of the resident, and number of comorbidities of the patient. We accounted the experience of being an auditor after a participant's first auditing experience; in other words, we set this variable as time-dependent. For example, if a resident functioned as an auditor in August, we coded any charts of that resident's patients audited before August as without the resident's auditing experience, but any chart audited after August as with his orher auditing experience. We categorized the duration of residents' training as less than 6 months, 6 to 18 months, and 19 months or longer, as the project extended over July 1, when promotion or matriculation occurred. The numbers of comorbidities excluded the diagnosis of interest. For example, if a patient had COPD and CHD, the number of comorbidities for the total score was two; however, the number of comorbidities for the diagnosis-specific score for COPD was one.
The differences of the total and diagnosis-specific scores by the factors of interest were tested by models. We chose univariate and multivariate generalized estimating equations (GEE) for modeling to account for three types of repeated measurements embedded in the study design: (1) a patient can be repeatedly measured using the 24 QI indices, (2) each resident had multiple medical records audited, and (3) each auditing resident reviewed multiple medical records. Models were constructed for the itemized QI index under the assumption of binomial distribution for the response with the link function of logit. The total score analysis used the weight for each diagnosis-related item as described above. The statistical tests for the significance of variation by the factors (month of clinic visit, prior experienceof being an auditor, duration of training of the resident, and number of comorbidities) were performed with these models using type 3 tests. To display the results from the multivariate GEE, the adjusted logits estimated from the model were converted to probabilities and multiplied by 100, which was interpreted as the mean score and ranged from 0 to 100. The adjustment was made by entering the mean values for the covariates other than the one of interest into the models. A probability of .05 for type I error was deemed to be statistically significant. All analyses were performed with SAS 9.1 for Windows (SAS Institute, Inc., Cary, North Carolina).
Audited medical records and participating residents
In the four months, we reviewed 493 medical records audited by residents. We excluded one medical record missing the auditing resident's name, one record missing a clinic resident's name, and one record audited by the resident who provided care. We further excluded 21 medical records which were repeatedly audited for the same patients. Therefore, 469 medical records were included in the analysis.
At least one medical record was audited for 80 out of 109 residents (73%). The characteristics of the residents and the medical records of their patients are shown in Table 2. Approximately 80% of the cases had one or two diagnoses of interest. A median of 4.5 (range 1–21) medical records of patients were audited for each resident. Twelve among 65 (18%) second- or third-year residents performed the audit. Each auditing resident reviewed a median of 21.5 (range 7–125) medical records.
Quality measurement indicator performance
The medical record audit yielded 441 total and 798 diagnosis-specific scores. Twenty-eight patients had none of the five selected diagnoses. The positive response rate for itemized indices varied widely (Table 1). The positive response rates for weight measurement (LVF2) and beta-blocker use (LVF5) were 100%. The three lowest positive response rates were 44% for influenza / pneumococcal vaccines (COPD3), 45% for neuropathy testing recorded (DM4), and 53% for spirometry and reversibility testing offered (COPD1). The mean total score for all QI indices was 89 (95% CI = 88–91). The diagnosis-specific scores varied: COPD 59 (95% CI = 49–69), CHD 84 (95% CI = 80–87), DM 78 (95% CI = 75–80), HTN 98 (95% CI = 96–99), and LVF 96 (95% CI = 94–98) (Table 3).
Trends of scores over time
The mean total score significantly varied by month. Scores in March, April, August, and September were 88 (95% CI = 85–91), 94 (95% CI = 90–96), 87 (95% CI = 85–89), and 91 (95% CI = 89–93), respectively (P = .002, Table 3). Scores seemed to improve between the first two consecutive months (March and April), and after the three-month gap between April and August, the scores significantly declined. This tendency was consistently observed for COPD, CHD, and DM scores.
To examine the possibility that the drop between April and August was attributable to a group of first-year residents who started their training in July, we examined the score for these months by duration of training (Figure 1), but this did not seem to be the case. As shown in the figure, the decline was recognized in the continuing residents with durations of training of 6 to 18 months and of 19 months or longer.
Scores by experience as an auditor
The total score of 58 records of residents who had served as auditors was 94 (95% CI = 89–96) compared with 89 (95% CI = 87–90) in their counterpart (411 records, P = .02, Table 3). A similar relationship exists for the DM score.
Scores by duration of training
The total score increased by duration of training (Table 3). The total score was 86 (95% CI = 83–89) for residents with less than 6 months of training and 91 (95% CI = 90–93) for residents with 19 months or more of training (P = .01). This trend was observed for diagnosis-specific scores of COPD, CHD, and DM, but only DM showed statistical significance (P = .0006). HTN and LVF scores were high regardless of duration of training.
Score by number of comorbidities
The total score was highest for single-diagnosis cases (97 [95% CI = 94–98]); the scores for cases with two or more comorbidities were significantly lower than the ones for single-diagnosis cases (P < .0001).
Using GEEs, the total adjusted scores were calculated and displayed in Table 4. The adjusted total scores varied significantly by month (P = .0009) as seen with univariate analyses. The scores improved between two consecutive months and then declined from April to August. The total adjusted score was significantly higher for single-diagnosis cases than for more complicated cases (P < .0009). Experience as an auditor and duration of training did not make any difference in the total adjusted scores. The adjusted diagnosis-specific scores showed similar trends as univariate analyses but were not statistically significant.
This report demonstrated the feasibility and effectiveness of a QI intervention in an urban university residency program, as well as relevant factors in such an intervention. The intervention used in this study is characterized by a limited number of didactic sessions, a medical record audit, and feedback with a motivational plan (unimplemented). This approach resulted in improvement in clinical QI indices. This is consistent with the conclusions suggested in a recent systematic review4: Most QI projects involving residents, summarized in the review, showed improvement in clinical outcomes, such as documentation and implementation of pertinent physical examinations, clinical laboratories, and vaccinations. Our intervention incorporated several adult-learning principles and major domains for improvement proposed by the Institute of Healthcare Improvement,23 including active participation of learners and feedback. This simple design allowed an intervention applicable to a nonacademic, real-life practice.
Our intervention favored the residents' active participation with auditing medical records over traditional didactic training. This approach demonstrated a significant increase in QI indices, which seemed predominant in COPD, CHD, and DM. Scores of HTN and LVF were close to 100 from the beginning; thus, their lack of significant improvement is likely due to a ceiling effect. Previous studies emphasized theoretical QI content with didactic training in a randomized trial undertaken in an HMO to optimize continuous QI24 and in a randomized trial based in primary care clinics undertaking diabetes care.25 These authors found little evidence of clinical benefit.24,25 As for QI initiatives in residency programs, another study introduced a four-week practice-based learning and improvement elective rotation with weekly didactic teaching.8 The result of this effort demonstrated the participants' lack of necessary understanding for making and sustaining changes in a system. An important feature of our intervention is feedback, a feature which has been shown to improve learners' performance.26 This approach might have worked more effectively if it had been combined with a motivational plan to overcome the phenomenon of “clinical inertia,” whereby physicians fail to initiate or intensify therapy when well-defined and accepted treatment goals are present.27
The simple design and implementation process made this project extendable and sustainable with minimal resources. Previously, pitfalls associated with QI interventions have included concern that such efforts may encroach on productivity of residents,11 demand faculty time,7,10,11 or require external funding.11 Though our project did not formally measure the cost of intervention, 2 one-hour didactic sessions and roughly two hours of medical record audit time per day by one resident could be considered relatively inexpensive.
Compliance with QI indices in our project was generally high with a few exceptions. Although we used slightly different QI indices, in comparison, our performance was slightly lower or similar to two previous studies in the United States21 and in England.28 For example, regarding diabetes management, our study showed that compliance with the measurements of HbA1c and serum creatinine was similar to the previous U.S. study,21 but we observed lower compliance in eye exams, screening for nephropathy, and foot examination. The lower compliance with eye exams and screening for nephropathy could be lack of awareness, whereas foot examination compliance could be due to a lack of time and may be consistent with the lack of immediate availability of a monofilament. Regarding coronary artery disease, our study showed equal compliance in aspirin use and higher in beta-blocker use but lower compliance with exercise testing or specialist assessment, and lower compliance with smoking cessation advice compared with a previous publication from England.28 The relatively low compliance with exercise testing or specialist assessment may mean that many patients with known coronary artery disease at the clinic are stable in symptoms. The lower compliance with smoking cessation advice could be from lower awareness or lack of time; however, encouraging smoking cessation is particularly important in our setting, because our state has one of the highest smoking rates in the United States.29 Taken together, our results indicate that many residents can achieve a high competency level in the clinical management of common diseases, but it also seems important to reinforce guidelines and establish a proper clinic environment.
We observed that QI scores significantly declined from April to August. An explanation may be that discontinuation of the intervention from May through July had a detrimental effect. This may suggest that continuing the QI intervention is necessary for a sustained effect, a finding reported by other investigators.10 It is unclear from this study which components of the intervention (i.e., didactic sessions, auditing, or feedback) would be most effective if we continued certain actions from May through July. It would be cost-effective if less intensive interventions, such as infrequent didactic sessions or reminding feedback, can help sustain the effects of intermittent intensive interventions.
Our study showed little difference in the measured QI indices between entry-level and advanced residents (Tables 3 and 4). Yet, cases with more comorbidities scored significantly lower in QI indices overall by all residents. Failure to offer high-quality health care may be a result of limited time in high-flow clinics. We may be able to achieve an improvement in quality simply by extending the time allocated for patients with multiple comorbidities, rather than extensive QI initiatives that would lessen time spent for the care of all patients. This hypothesis may partially explain the failure of the previous two trials to achieve their projected clinical goals.24,25
Although this research study carries the aforementioned strengths in the intervention, four limitations are worth noting. First, the effectiveness of the intervention was measured by trends of QI indices over time and by the comparison of the experience of the resident as an auditor; that is, the study design did not allow us to randomize residents as participants or residents as auditors. However, using a controlled design, or even a randomized controlled design, is not devoid of problems,30 as there can be difficulty in recruiting physicians to participate and including “atypical” clinic group practices.25 Although we were not aware of any significant changes in external conditions, we cannot exclude the possibility that the changes in QI indices over time were due to external conditions rather than the intervention. Second, our approach did not involve residents in the planning of this project. Perhaps such involvement could have enhanced their interest in future participation in QI projects.31 Third, the calculated composite scores incorporated “process parameters” and “clinical parameters.” The improvement in QI performance may reflect a “process” improvement and better documentation, not necessarily an outcome improvement. Finally, the duration of the intervention was relatively brief, and the long-term impact of this intervention is not clear.
In conclusion, the residents' compliance with standard care at their continuity clinic was generally high in response to this QI project with a limited number of didactic sessions, a medical record audit, and feedback. The residents' performance varied according to diagnosis and by the complexity of cases. Continuity of the audit process may be necessary for a sustained effect on QI. The rather simple QI approach undertaken in this study can be useful in the context of graduate medical education.
2 Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001.
3 Berwick DM. Continuous improvement as an ideal in health care. N Engl J Med. 1989;320:53–56.
4 Boonyasai RT, Windish DM, Chakraborti C, Feldman LS, Rubin HR, Bass EB. Effectiveness of teaching quality improvement to clinicians: A systematic review. JAMA. 2007;298:1023–1037.
5 Ashton CM. “Invisible” doctors: Making a case for involving medical residents in hospital quality improvement programs. Acad Med. 1993;68:823–824.
6 Ellrodt AG. Introduction of total quality management (TQM) into an internal medicine residency. Acad Med. 1993;68:817–823.
7 Djuricich AM, Ciccarelli M, Swigonski NL. A continuous quality improvement curriculum for residents: Addressing core competency, improving systems. Acad Med. 2004;79:65–67.
8 Ogrinc G, Headrick LA, Morrison LJ, Foster T. Teaching and assessing resident competence in practice-based learning and improvement. J Gen Intern Med. 2004;19:496–500.
9 Baker R, Green L. Quality assurance for family practice trainees in the USA and the UK: Too little too late? Med Educ. 1990;24:258–263.
10 Weingart SN. A house officer-sponsored quality improvement initiative: Leadership lessons and liabilities. Jt Comm J Qual Improv. 1998;24:371–378.
11 Nuovo J, Balsbaugh T, Barton S, et al. Development of a diabetes care management curriculum in a family practice residency program. Dis Manag. 2004;7:314–324.
12 Fox CH, Mahoney MC. Improving diabetes preventive care in a family practice residency program: A case study in continuous quality improvement. Fam Med. 1998;30:441–445.
13 Parenti CM, Lederle FA, Impola CL, Peterson LR. Reduction of unnecessary intravenous catheter use. Internal medicine house staff participate in a successful quality improvement project. Arch Intern Med. 1994;154:1829–1832.
15 Rosenthal MB, Landon BE, Normand SL, Frank RG, Epstein AM. Pay for performance in commercial HMOs. N Engl J Med. 2006;355:1895–1902.
16 Physician payment and quality improvement. P L 109-432, Approved December 20, 2006 (120 Stat 2922) Tax Relief and Health Care Act of 2006. 2006; Division B-Medicare and other health provisions (Medicare Improvements and Extension Act of 2006).
17 Holmboe ES, Wang Y, Meehan TP, et al. Association between maintenance of certification examination scores and quality of care for medicare beneficiaries. Arch Intern Med. 2008;168:1396–1403.
18 Shea JA, Norcini JJ, Kimball HR. Relationships of ratings of clinical competence and ABIM scores to certification status. Acad Med. 1993;68(10 suppl):S22–S24.
19 Tamblyn R, Abrahamowicz M, Dauphinee WD, et al. Association between licensure examination scores and practice in primary care. JAMA. 2002;288:3019–3026.
20 Doran T, Fullwood C, Gravelle H, et al. Pay-for-performance programs in family practices in the United Kingdom. N Engl J Med. 2006;355:375–384.
21 Mangione CM, Gerzoff RB, Williamson DF, et al. The association between quality of care and the intensity of diabetes disease management programs. Ann Intern Med. 2006;145:107–116.
22 Hunt SA; American College of Cardiology; American Heart Association Task Force on Practice Guidelines (Writing Committee to Update the 2001 Guidelines for the Evaluation and Management of Heart Failure). ACC/AHA 2005 guideline update for the diagnosis and management of chronic heart failure in the adult: A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Writing Committee to Update the 2001 Guidelines for the Evaluation and Management of Heart Failure). J Am Coll Cardiol. 2005;46:e1–e82.
23 Batalden P, Berwick D, Bisognano M, Splaine M, Baker G, Headrick LA. Knowledge Domains for Health Professional Students Seeking Competency in the Continual Improvement and Innovation of Health Care. Boston, Mass: Institute of Healthcare Improvement; 1998.
24 Solberg LI, Kottke TE, Brekke ML, et al. Failure of a continuous quality improvement intervention to increase the delivery of preventive services. A randomized trial. Eff Clin Pract. 2000;3:105–115.
25 O'Connor PJ, Desai J, Solberg LI, et al. Randomized trial of quality improvement intervention to improve diabetes care in primary care settings. Diabetes Care. 2005;28:1890–1897.
26 Boekeloo BO, Becker DM, Levine DM, Belitsos PC, Pearson TA. Strategies for increasing house staff management of cholesterol with inpatients. Am J Prev Med. 1990;6:51–59.
27 Baruch L, Phillips RA. Quality: The need for intelligent efforts. Arch Intern Med. 2005;165:1455–1456.
28 Campbell S, Reeves D, Kontopantelis E, Middleton E, Sibbald B, Roland M. Quality of primary care in England with the introduction of pay for performance. N Engl J Med. 2007;357:181–190.
29 Centers for Disease Control and Prevention (CDC). State-specific prevalence of current cigarette smoking among adults and secondhand smoke rules and policies in homes and workplaces—United States, 2005. MMWR Morb Mortal Wkly Rep. 2006;55:1148–1151.
30 Prideaux D. Researching the outcomes of educational interventions: A matter of design. RTCs have important limitations in evaluating educational interventions. BMJ. 2002;324:126–127.
31 Volpp KG, Grande D. Residents' suggestions for reducing errors in teaching hospitals. N Engl J Med. 2003;348:851–855.