Skip Navigation LinksHome > October 2004 - Volume 79 - Issue 10 > A Call for Outcomes Research in Medical Education
Academic Medicine:
Special Theme Article

A Call for Outcomes Research in Medical Education

Chen, Frederick M. MD, MPH; Bauchner, Howard MD; Burstin, Helen MD, MPH

Free Access
Article Outline
Collapse Box

Author Information

Dr. Chen is a Kerr White Visiting Scholar, and Dr. Burstin is director, Center for Primary Care, Prevention, and Clinical Partnerships, Agency for Healthcare Research and Quality, Rockville, Maryland; Dr. Bauchner is professor, Division of General Pediatrics, Boston University School of Medicine, Boston Medical Center, Boston, Massachusetts.

Correspondence should be addressed to Dr. Chen, Department of Family Medicine, University of Washington, Box 354982, Seattle, Washington 98195; telephone: (206) 685–0402; e-mail: 〈fchen@u.washington.edu〉.

Collapse Box

Abstract

The primary goal of medical education is to produce physicians who deliver high-quality health care. Recent calls for greater accountability in medical education and the development of outcomes research methodologies should encourage a new research effort to examine the effects of medical training upon clinical outcomes. The authors offer a research agenda that links medical education and quality of health care and give specific examples of potential research projects that would begin to examine that relationship. A proposed model of patient outcomes research in medical education recognizes the contributory effects of health care system-level factors as well as the continuum of medical education, process measures, and individual training and preparedness to deliver high-quality care. There exists an opportunity to create a research agenda in medical education outcomes research that is multidisciplinary, broad based, and focused on patient-centered outcomes.

The primary goal of medical education is to produce physicians who deliver high-quality health care. To achieve this goal, medical education encompasses a broad continuum of distinct and diverse educational processes from medical school to residency training to continuing medical education. There has been, however, remarkably little investment into the conceptualization and study of the association between the process of medical education and quality of care. In this article, we propose a research agenda that links medical education and quality of health care and offer specific examples of proposed research projects that would begin to examine that relationship. We also propose a theoretical model that may help demonstrate the complex relationships between medical education and patient-level outcomes.

Back to Top | Article Outline

Background

Recent calls for accountability and definition of the return-on-investment for medical education have generated new interest in examining the connection between medical education and clinical outcomes.1,2 As a recent Commonwealth Fund Task Force report emphasized, “the quality of care that the public receives is determined to some extent by the quality of medical education students and residents receive.”3

Historically, the authority and autonomy for medical education and training have been delegated to the medical profession. Medical education was assumed to be an intrinsic part of the profession's capacity to self-regulate. As a result, there has been limited impetus for the educational research community to examine the patient-level outcomes of medical education.4 The research enterprise in medical education has been primarily focused on educational, rather than clinical, outcomes.2,5 A recent review of 600 articles published in medical education research journals revealed only four studies that measured clinical outcomes of patients.6 The remainder were divided between measuring learners’ acquisition of knowledge and their satisfaction.

The anemic funding of medical education research has offered little incentive to devise creative studies that investigate the effects of medical education. In addition to the paucity of readily available datasets to examine the performance of medical school graduates, there are formidable biases and confounders that distort the effect of education upon physicians’ performance. The latency of educational effect, individual variations, the difficulty of controlling any educational intervention, and the overarching role of the practice and care system potentially dilute any ability to measure outcomes.7–9 The predominant emphasis of medical education research on undergraduate education has focused investigators on short-term outcomes that avoid the complexities of residency programs and subsequent clinical practice. Finally, little investment has been made into developing a theory of medical education and its influence on outcomes.10

Previous efforts to examine medical education outcomes have gained little traction. In the late 1980s, Wartman and O'Sullivan11 called for the creation of a national center for health professions education research. (In this issue of Academic Medicine, Wartman presents his latest thoughts on such a center). Four years later, an agenda-setting conference for medical education research highlighted the need to study the effects of medical education on patient outcomes.12 Although the conference resulted in the creation of a few centers for medical education research, their work focused primarily on workforce supply and specialty choice. The call for “a research program to demonstrate the efficacy of medical education in influencing practice outcomes” went largely unheeded.12

On the other hand, the public investment in medical education cannot be overlooked. In 2000, Medicare payments for graduate medical education (GME) totaled almost $8 billion and supported over 100,000 medical interns and residents. These subsidies comprise both direct payments to hospitals for residents’ and faculty salaries, and indirect payments for the added patient care costs associated with teaching.13,14 Although the Balanced Budget Act of 1997 reduced payments in GME to teaching hospitals, recent efforts to increase indirect payments have focused attention on the entire GME financing mechanism.15,16 At the undergraduate medical education level, Hopkins reported that over three-quarters of the nation's 125 medical schools received public subsidies estimated to be in excess of $2 billion in the year 2000.17

The current emphasis on quality improvement and patient safety in medical care also provides a new opportunity for the reexamination of the role of medical education. As studies document variations in quality of care and prescribing habits, as well as the prevalence of medical errors, little attention has been focused on the source of physicians’ attitudes, skills, and knowledge about these issues. While the quality movement has been rightly focused on changes in health care systems, the influence of medical education on individual performance, quality, and physicians’ roles while working within systems remains obscure.18 Two studies (one reported in 1998, the other in 2002) have demonstrated that differences in quality of care can in fact be attributed to individual physicians’ education, certification, and performance.19,20

Current efforts also suggest that the medical education community is prepared to examine the link between quality of care and medical education. The formation of the best-evidence medical education movement, culminating in the international Campbell Collaboration, represents an effort to apply evidence-based medicine criteria to medical education interventions.21,22 The Medical School Outcomes Project of the Association of American Medical Colleges and the Outcome Project of the Accreditation Council for Graduate Medical Education (ACGME) both reflect a new appreciation of the need to examine medical training and ensure the quality of the graduates of medical education programs.23,24

The scale of investments in medical education, the emerging calls for greater accountability in medical education, and more effective methodologies and outcomes research have changed the environment for research in medical education. The 2001 Institute of Medicine's report, Crossing the Quality Chasm: A New Health System for the 21st Century, highlighted the need for medical education and workforce training to be reoriented to address health care quality and to develop strategies for restructuring clinical education to fit 21st century health care needs.18 One recommendation focused on studying the link between quality outcomes and training. The report recognized that the foundation for quality health care rests upon developing good communication skills, interdisciplinary collaboration, evidence-based medicine, tools to manage knowledge, and shared decision making, across a full range of care settings.

Ultimately, the momentum to study clinical outcomes of medical education must come from within the medical education community. One of the abiding truths of medical education is that evaluation drives curriculum. With the pressing need to improve the quality of health care and public calls for accountability, the medical education research effort will be challenged to develop evaluations and outcome measures that satisfy these needs and motivate curricular change to improve the health of patients.

Back to Top | Article Outline

Research Agenda

The Agency for Healthcare Research and Quality (AHRQ) and the Bureau of Health Professions of the Health Resources and Services Administration convened an expert meeting to discuss medical education outcomes research in 2002. Thirty national experts and opinion leaders from the fields of medical education and health services research participated with the intention to open a dialogue between medical education researchers, outcomes researchers, and stakeholder organizations about the measurement of clinical outcomes of medical education. Three critical questions were posed:

* How do we demonstrate that medical education is producing physicians who deliver high-quality care?

* What is the effect of medical education on improving patient care?

* What is the potential for research using patient-centered clinical outcomes to measure the performance of medical education?

Specific guidance was given to include both undergraduate and graduate medical education in the discussion. Participants were charged with considering both the explicit and “hidden” curricula in medical education and training programs.25 The ultimate goal was to examine patient-level, clinical outcomes, in addition to important intermediate process measures.

Below we summarize the expert meeting's discussion of a research agenda that links medical education and quality of health care. We also describe specific examples of proposed research projects that would begin to examine that relationship. We then offer a theoretical model that may help show the complex relationships between medical education and patient-level outcomes.

Back to Top | Article Outline
Medical Education and Quality Improvement

Keeping up with the rapid pace of change in the health care system has dictated that medical trainees learn about ways to improve the quality of care over the course of their careers. Current training methods do not orient physicians to practice collaboratively or with self-reflection. There is little emphasis on learning about continuous quality monitoring and improvement. This situation is changing with the implementation of two of the ACGME’s “competencies,” Systems-Based Practice and Practice-Based Learning and Improvement.26 These requirements reflect the critical linkage between medical education, learning the skills of quality improvement, and the actual application of those lessons to improve health care for patients.

There is an opportunity and need for research into the effect of medical education on individual physicians’ ability to change and adapt, as well as their ability to improve practice outcomes. Improvement in patient outcomes is often linked to the ability of physicians to change and adopt new practices within their care settings. There is particular interest in learning whether training in health care system improvement actually results in physicians’ being more likely to effect positive changes in their clinical settings. For example, can physicians who are trained in a delivery system oriented to quality improvement bring those attitudes and skills to a different type of care system? Do residents who are trained in hospitals with highly developed medical informatics systems demand or introduce computerized order-entry and decision-support systems into their subsequent clinical settings? Does the quality of care in those settings improve?

Other potential studies include identifying high- and low-performing physicians and retrospectively examining the elements of their medical education, self-learning, and self-assessments. Residency programs that use interdisciplinary care teams should demonstrate care improvements and be contrasted with residency programs that do not use interdisciplinary care teams. Medical schools that teach lifelong learning, reflective practice, and integrative thinking can be studied to show whether they are more likely to produce physicians who are able to improve their patients’ care.

Back to Top | Article Outline
Challenges and Strategies

Research on the outcomes of medical education has great potential but is fraught with methodological challenges. Among the fundamental difficulties in doing meaningful medical education outcomes research are the complexity and number of variables and confounders that can bias any association between education and clinical outcomes. The considerable lag time between an educational intervention and the actual measurement of patient outcomes also constitutes a formidable barrier. The likelihood of a significant effect from a myriad of other confounding factors, such as the clinical care system or comorbidity, is ever-present. System-level variables such as policies and regulations may influence patient care outcomes. The wide variation among and between training programs and medical schools will also need to be taken into account. Recently developed statistical methods will have to be explored to help estimate these effects and account for them in analyses. These methods may include bootstrapping, path analysis, hierarchical analysis, and generalizability theory.

Another way to account for these challenges is to measure broad-based outcomes, utilizing both qualitative and quantitative methods. Medical education outcomes research will need to involve a convergence of medical education researchers and patient outcomes researchers. This research will require a multidisciplinary approach that engages medical education researchers, clinicians, health services researchers, anthropologists, sociologists, and economists. In the past, medical education researchers have rarely collaborated with health services researchers. These connections can be made through collective research seminars and the mutual recognition of the role that medical education plays in health care quality. In addition, rigorous multi-institutional studies, including randomized controlled trials and other novel designs, will be needed in order to compare educational programs. Only through multimethod assessments of quality and performance can the contributing effects of practice systems and training programs on individual data be clarified.

Back to Top | Article Outline

Proposed Theoretical Model

The complex relationships among medical education, process measures and determinants, and patients’ health outcomes can be described using a conceptual model (see Figure 1). The first level recognizes the continuum of medical education from undergraduate medical education (UME) to GME to continuing medical education (CME). While these represent distinct phases and environments of medical education, all contribute to a physician's development in different dimensions—knowledge, skills, behaviors, and attitudes. In turn, the interactions of these dimensions with a particular health care system create patient health outcomes. Each level of this model deserves some attention.

Figure 1
Figure 1
Image Tools
Back to Top | Article Outline
Continuum of Medical Education

The continuum of medical education encompasses very different educational interventions, styles, and environments. Because of this, research efforts have traditionally been situated in one of the three distinct educational domains: UME, GME, or CME. However, the linkages between these three domains should be clarified, since they all contribute differently to physicians’ abilities and the subsequent health care delivered.

UME tends to be knowledge-focused, taking undifferentiated learners and teaching them knowledge and basic clinical skills. One of the challenges of UME is how to teach information management and mastery in an increasingly evidence-based system. GME learners, in contrast, are better differentiated, having already committed to a particular medical specialty. The educational model of GME is much more one of apprenticeship. While these distinctions are well established, the differential effects of the medical education continuum have not been explored. For example, there is little consensus as to the relative influence of UME compared to that of GME. The perception that GME experiences are more influential than those of UME needs to be examined. Whether physicians’ practice styles are more heavily molded by their clinical training or by their medical school education remains unanswered. The increasing degrees of physicians’ differentiation as they progress through specialty training may also require different types of measurement. The evaluation and outcomes measures for a gastroenterologist, for instance, may be quite different than those for a family physician. Finally, more is known about CME, as its educational interventions tend to be more discretely contained. However, clear effects of CME on patient outcomes are still difficult to demonstrate.2,27

The training of physicians to provide diabetes care offers a model of education that leads to improved health outcomes. Physicians’ education about diabetes begins in the earliest years of medical school in anatomy, physiology, and pathology. That education becomes applied in the clinical years as medical students learn to care for diabetic patients in inpatient and outpatient settings. In GME, residents may learn about diabetes care through hands-on clinical care as well as through formal teaching in didactics and bedside rounds. The role of CME curricula in diabetes is also manifest as new medications and techniques are introduced into diabetes care.

Back to Top | Article Outline
Process Measures

There is a vast body of literature about the second level of the medical education model—the development of knowledge, specific clinical skills, behaviors, and attitudes. These measures represent processes that are critical steps in the pathway to creating the quality of care that can be measured by patient-level outcomes. The testing of knowledge comprises many of the traditional measures of educational success. These measures include board certification, knowledge testing, and the evaluation of specific clinical skills, and are often used to measure the success of educational interventions.28 While process measures like these certainly have utility, the association between them and clinical quality of care must be made explicit. It is particularly concerning that this relationship has not always been positively demonstrated.28,29

For example, research into the relationship between clinical outcomes and the recently implemented ACGME core competencies is needed.30 Even such laudable goals such as lifelong learning, patient-centeredness, and the core competencies are process measures that hope to translate into improved patient care, but the data linking these measures to outcomes are scant. Similarly, the development of objective structured clinical examinations and standardized patients are novel evaluation tools but have not been examined with respect to their ability to predict actual clinical practice and patient outcomes.

Physicians’ skills and behaviors are intermediate measures that may offer a closer association with patient outcomes. Medical education researchers should focus on identifying the set of modifying determinants that directly affect clinical outcomes. These include prescribing patterns, the use of counseling and shared decision making, adherence to practice guidelines, and provision of culturally appropriate care. These determinants may also include individual behaviors that act at levels beyond the clinical encounter, such as the ability to change behavior and clinical practice based on new evidence, the ability to work in teams, and the tendency to self-evaluate.

Using the diabetes example, educational events along the UME-GME-CME continuum contribute to physicians’ knowledge, skills, and behavior in diabetes care. The knowledge outcomes of medical education may include understanding the pathophysiology, clinical signs and symptoms, diagnosis, and ideal management of diabetes. The skill set around diabetes may include appropriate prescribing, glycosylated hemoglobin monitoring, or preventive counseling for optimal diabetes care. Physicians’ behavioral outcomes may include patient-centeredness or the use of shared decision making in determining a patient's ideal diabetes treatment regimen. Again, there is a significant amount of literature that has already demonstrated the effect of education on many of these intermediate process outcomes. The critical focus of research should be on the subsequent link between these process outcomes and patient-level clinical outcomes.

Back to Top | Article Outline
Patient Outcomes

Care should be taken to identify appropriate and measurable patient health outcomes when examining the effect of medical education. Too narrow a set of outcomes will promote bias towards achieving only those measures. Crossing the Quality Chasm offers a potential framework for medical education outcomes. That report calls for measures that demonstrate the principles of safe, effective, patient-centered, timely, efficient, and equitable health care.18 While these measures have been used to gauge the performance of health care delivery systems, a similar framework could be applied to the outcomes of training individual providers.31

Using the example of diabetes, clinical outcome measures may include improved glycemic control, fewer episodes of hypoglycemia, prevention of complications, and ultimately, reduced mortality. In diabetes, physicians’ particular skills and behaviors, in conjunction with the clinical care system, result in better health outcomes for patients. The model demonstrates the influence of the continuum of medical education on physicians’ abilities, and recognizes the ultimate effect of that education on patients’ health.

Back to Top | Article Outline
Health Care System Factors

Encompassing the arena of medical education and outcomes is the penumbra of health care system factors that facilitate and modulate the delivery of high-quality medical care. The environmental factors, clearly delineated in Crossing the Quality Chasm, include evidence-based medicine, information technology, the professional workforce, and payment systems.18 (See Figure 1.) While much attention has since focused on the critical role of these factors in delivering high-quality care, the responsibility of the individual provider and the role of that provider's medical education and training must not be overlooked. Our model of patient outcomes research in medical education recognizes the contributory effects of both system-level factors and individual training, education, and preparedness to deliver high-quality care.

In the diabetes example, health care system characteristics clearly have an effect on quality of care in diabetes. The use of patient registries, nurse follow-up for diabetes care, and computerized decision support systems can modulate physicians’ behaviors and affect diabetes outcomes.

Back to Top | Article Outline

Next Steps

With growing calls for accountability in health care, it is critical to catalyze research that examines the linkages between medical education and quality health care. The recent Commonwealth Fund report, Training Tomorrow's Doctors: The Medical Education Mission of Academic Health Centers, found that “the available data are insufficient to judge the performance of academic health centers in discharging their educational responsibilities beyond establishing a minimum level of competency.”3 The report recommended that the federal government support research to produce valid and reliable measures of the costs and quality of medical education, specifically requesting $25 million in public funding to develop and implement improved measures of performance in medical education.

The pressing need to demonstrate the ability of medical education to improve health care requires the development of a research agenda around a model of the continuum of medical education that determines physicians’ knowledge, skills, and behaviors and ultimately affects patient health outcomes. Importantly, there is a need for research to examine not just the independent components of model, but also the critical linkages between the levels of the model.

A commitment to multidisciplinary, broad-based, intermediate, and patient-centered outcomes is necessary. A consortium of medical schools that have a vested interest in examining their own undergraduate and graduate medical education programs could lead the movement in medical education outcomes research. These schools’ academic health centers, taking advantage of their existing health services and medical education research resources, could take vital steps towards rigorously examining and ultimately improving medical education to improve health care.

The most compelling argument for the research agenda in medical education is the impending challenges for health care in the coming decades. The increasing age of the population, changes in illness patterns, and the evolution of technology and care delivery mandate an investment into the critical processes and outcomes of medical education. In our efforts to improve the quality and outcomes of health care, it is critical that we weave medical education back into the fabric of systems improvement. As stated in Crossing the Quality Chasm, “Health care is not just another service industry…. The people who deliver care are the health system's most important resource.”18 We in academic medicine need to ensure that the education that we are providing for medical students and residents and physicians in practice will prepare them to meet the challenge of achieving an accountable, high-quality health care system for the 21st century.

The views expressed are those of the authors, and no official endorsement by the Agency for Healthcare Research and Quality or the Department of Health and Human Services is intended or should be inferred. The authors would like to thank the participants in the AHRQ Expert Meeting on Medical Education Outcomes Research for their thoughtful contributions.

Back to Top | Article Outline

References

1.Phillips J. Return on Investment in Training and Performance Improvement Programs. Houston: Gulf, 1997.

2.Gilman SC, Cullen RJ, Leist JC, Craft CA. Domains-based outcomes assessment of continuing medical education: the VA model. Acad Med. 2002;77:810–7.

3.The Commonwealth Fund Task Force on Academic Health Centers. Training Tomorrow's Doctors: The Medical Education Mission of Academic Health Centers. New York: The Commonwealth Fund, 2002.

4.Whitcomb ME. Research in medical education: what do we know about the link between what doctors are taught and what they do? Acad Med. 2002;77:1067–8.

5.Norman G. Research in medical education: three decades of progress. BMJ. 2002;324:1560–2.

6.Prystowsky JB, Bordage G. An outcomes research perspective on medical education: the predominance of trainee assessment and satisfaction. Med Educ. 2001;35:331–6.

7.Prideaux D. Researching the outcomes of educational interventions: a matter of design. BMJ. 2002;324:126–7.

8.Tamblyn R. Outcomes in medical education: what is the standard and outcome of care delivered by our graduates? Adv Health Sci Educ Theory Pract. 1999;4:9–25.

9.Bordage G, Burack JH, Irby DM, Stritter FT. Education in ambulatory settings: developing valid measures of educational outcomes, and other research priorities. Acad Med. 1998;73:743–50.

10.Wartman SA. Research in medical education: the challenge for the next decade. Acad Med. 1994;69:608–14.

11.Wartman SA, O'Sullivan PS. The case for a national center for health professions education research. Acad Med. 1989;64:295–9.

12.Proceedings of the BHPr–AAMC Conference. Research in Medical Education: Policies for the Future. Fairfax, Virginia, September 19–21, 1993. Acad Med. 1994;69:601-27.

13.Iglehart JK. Support for academic medical centers: revisiting the 1997 Balanced Budget Act. N Engl J Med. 1999;341:299–304.

14.Iglehart JK. Medicare and graduate medical education. N Engl J Med. 1998;338:402–7.

15.Gbadebo AL, Reinhardt UE. Economists on academic medicine: elephants in a porcelain shop? Health Aff (Millwood). 2001;20:148–52.

16.Newhouse JP, Wilensky GR. Paying for graduate medical education: the debate goes on. Health Aff (Millwood). 2001;20:136–47.

17.Hopkins M. Subsidies to Penn State's Medical Program Lag behind National Rivals. Centre Daily Times. October 13, 2002:6A.

18.Institute of Medicine. Crossing the quality chasm: a new health system for the 21st century. Washington, DC: National Academy Press, 2001.

19.Prystowsky JB, Bordage G, Feinglass JM. Patient outcomes for segmental colon resection according to surgeon's training, certification, and experience. Surgery. 2002;132:663–70.

20.Tamblyn R, Abrahamowicz M, Brailovsky C, et al. Association between licensing examination scores and resource use and quality of care in primary care practice. JAMA. 1998;280:989–96.

21.Harden RM, Lilley PM. Best evidence medical education: the simple truth. Med Teach. 2000;22:117–9.

22.Wolf FM, Shea JA, Albanese MA. Toward setting a research agenda for systematic reviews of evidence of the effects of medical education. Teach Learn Med. 2001;13:54–60.

23.Whitcomb ME. Competency-based graduate medical education? Of course! But how should competency be assessed? Acad Med. 2002;77:359–60.

24.Stevens DP. Three questions for the LCME. Acad Med. 2000;75:960–1.

25.Cohen JJ. Our compact with tomorrow's doctors. Acad Med. 2002;77:475–80.

26.ACGME Outcome Project: General Competencies 〈http://www.acgme.org/outcome/comp/compHome.asp〉. Accessed 23 June 2004.

27.Mazmanian PE, Davis DA. Continuing medical education and the physician as a learner. JAMA. 2002;288:1057–60.

28.Sharp LK, Bashook PG, Lipsky MS, Horowitz SD, Miller SH. Specialty board certification and clinical outcomes: the missing link. Acad Med. 2002;77(6):534–42.

29.Mant J, Hicks N. Detecting differences in quality of care: the sensitivity of measures of process and outcome in treating acute myocardial infarction. BMJ. 1995;311:793–6.

30.Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus S. General competencies and accreditation in graduate medical education. Health Aff (Millwood). 2002;21:103–11.

31.Institute of Medicine. Guidance for the National Healthcare Disparities Report. Washington, DC: National Academy Press, 2002.

© 2004 Association of American Medical Colleges

Login

Article Tools

Images

Share