Sharp, Lisa K. PhD; Bashook, Philip G. EdD; Lipsky, Martin S. MD; Horowitz, Sheldon D. MD; Miller, Stephen H. MD, MPH
Board certification by an American Board of Medical Specialties' (ABMS') member board has become the de facto standard by which the profession and the public recognize physician specialists in the United States. As of 1998, the 24 member boards of the ABMS had certified approximately 87% of the more than 635,000 licensed physicians in the United States.1 Currently, the boards issue certificates in 36 general specialties and 88 subspecialties.2 Most hospitals, managed care organizations, and health insurance plans require board certification for physicians wishing to obtain clinical privileges and join provider panels. The two largest organizations that accredit hospitals and other health-care-provider organizations, the Joint Commission on Accreditation of Healthcare Organizations3 and the National Committee for Quality Assurance,4 incorporate board certification into their accreditation standards. In addition to the 24 ABMS member boards, approximately 180 non-ABMS boards issue specialty certificates, suggesting that physicians perceive a need to be certified. The public also uses board certification as a measure of a physician's expertise, despite well-documented statements by the ABMS and the member boards that board certification is but one of several qualifications to be considered in assessing the quality of a physician's clinical care.5
Board certification is reasonably assumed to provide a measure of quality, since empirical evidence links measures of clinical care and measures of clinical knowledge and training, which are used to determine board certification. For example, higher scores on certification examinations correlate with measures of better patient care,6,7 and ratings in training correlate with clinical knowledge.8 However, despite the presumed link between certification and better clinical outcomes, no comprehensive review exploring the relationship between clinical outcomes and board certification has been published.
In this study we examined the published medical literature in the United States for evidence defining the relationship between clinical outcomes and board certification. Specifically, we questioned whether board certification by one of 36 general specialties recognized by the ABMS correlated either positively or negatively with clinical outcomes defined as accepted national standards of care.
A board-certified specialist in this study was defined as a physician with one or more valid certificates in the 36 general specialties recognized by the ABMS.2 Certification by an ABMS member board requires the physician “to successfully complete an approved educational program” accredited by the Accreditation Council for Graduate Medical Education (ACGME) and “pass a rigorous examination process administered by a member board, that is designed to assess the knowledge, skills and experience required to provide quality patient care in the specialty.”9 Clinical outcomes were defined as accepted national standards of care similar to those reported in the National Guideline Clearinghouse (NGC). The NGC is sponsored by the Agency for Healthcare Research and Quality, in partnership with the American Medical Association (AMA) and the American Association of Health Plans.
We conducted a systematic search for studies published between 1966 and July 1999 in three databases: OVID—Medline, psychological abstracts (PsycLit), and the Educational Research Information Clearinghouse (ERIC). The search consisted of the Medical Subject Headings (MeSH) and text words “certification, medical specialty, board certification, outcomes, clinical outcomes, and quality of health care.” In addition, we conducted 36 separate searches using the names of the 36 individual primary specialties approved by the ABMS as text words paired with the text word “certification.” All search results were limited to the English language. We manually searched the reference sections of each article for additional studies; one of us (LKS) and a doctorate-level researcher conducted independent searches.
Papers selected for review underwent three sets of screens, and we employed standard procedures for literature reviews.10 One of us (LKS) and a doctorate-level research assistant independently reviewed all abstracts in the initial selection of articles. When reviewers could not agree after reviewing an abstract or when references did not include an abstract, the full paper was obtained and reviewed. Papers were selected based on the review of the abstracts if they met these criteria (Criterion Set 1):
1. Measured at least one clinical outcome or process outcome
2. Mentioned board certification in the abstract or paper
Those papers meeting the initial screening criteria were selected for further study. We reviewed five studies independently to define criteria for further review of studies. Following the independent review, we met to obtain a consensus. Information was extracted from the remaining set of papers to judge studies against the next set of criteria (Criterion Set 2):
1. Reported on the clinical care provided by physicians in the United States
2. Compared at least one clinical outcome or process outcome for physicians having different certification qualifications or different specialty training
3. Defined and reported outcomes data as any measurable indicator of patient care, including mortality, morbidity, specific health outcomes, patient evaluation of care, costs for services, and malpractice litigation—studies measuring physicians' knowledge only through standardized tests or written examinations were excluded
4. Reported specialty board certification status as a variable in the data analysis
We independently reviewed the data extracted from the studies during the second screen for methodologic quality according to other criteria (Criterion Set 3):
1. Outlined specific criteria for selecting patients and physicians
2. Verified physician's specialty board certification using the ABMS database, member boards' databases, or derivative sources (e.g., the AMA Masterfile)
3. Analyzed data on clinical outcomes with comparisons for individual physician specialists
4. Based clinical outcomes on nationally recognized standardsPapers were evaluated on, but not initially excluded for failing to meet, this criterion (also a part of Criterion Set 3):
5. Used case-mix adjustments at any level (e.g., demographics, disease state)
The computer-based literature searches of three bibliographic databases identified 1,204 papers that mentioned any of the initial MeSH or text words in the abstract or title. Screening of the 1,204 abstracts identified 230 papers that measured at least one clinical or process outcome and mentioned board certification (Criterion Set 1). We manually searched the reference sections of all 230 articles to identify seven additional papers meeting the initial criteria. These 237 papers represented the universe of studies relevant to the research question. Of the 237, 56 (24%) measured American physicians' clinical outcomes, compared outcomes with certification status, and reported certification status as a variable in the data analysis (Criterion Set 2). We reviewed data from the 56 papers for quality of research method, verification of certification status, data-analysis methods, and application of nationally recognized standards of care for assessing outcomes (Criterion Set 3). We agreed that 13 papers in the data set (5%) with 33 separate findings met the screening criteria for relevance of an association between board certification status and clinical outcomes.
We excluded 43 of the 56 papers reviewed with Criterion Set 3 for various reasons. Twenty-two studies either did not verify board-certification status or used unreliable sources. If certification status is not confirmed, it is impossible to interpret in any meaningful way differences in clinical outcomes related to certification status. Another 20 articles confirmed board-certification status of the physicians using reliable sources but were excluded because of research design issues. For example, two studies combined board-certified physicians with “board-eligible” physicians.11,12 “Board-eligible” is not equivalent to board-certified, and frequently, this term describes physicians who have failed the certification process or completed ACGME accredited residency training but have not applied for certification.9 One study was excluded because the outcome variable was not a nationally accepted guideline for care.13
Of the 13 papers with 33 findings, two papers14,15 did not make case-mix adjustments, leaving in doubt the significance of four findings. We did not include these findings in the final analysis, reducing the number of findings to 29 reported in 11 papers. For the 29 remaining findings, 16 demonstrated positive and statistically significant associations between certification status and superior outcomes, and 13 demonstrated no evidence of an association. Excluding the two papers that failed to adjust for case mix, no evidence existed of worse outcomes related to certification. In all, four papers reported only positive findings, and four demonstrated a mix of positive findings and no evidence of an association. Table 1 lists these 13 papers, which are grouped into three clusters based on how the results were reported: (1) individual specialties, (2) multiple specialties grouped together, and (3) malpractice and licensure databases. For reference the two papers without case-mix adjustments are included.
The first cluster of five papers in Table 1 concerns individual specialties. Heck and colleagues16 compared board-certified and non-certified orthopedic surgeons' performances on knee replacements for severe osteoarthritis and found no association with certification status. The study was limited by the fact that 41 surgeons were board-certified, compared with only seven who were not. Kelly and Hellinger17,18 conducted two studies based on a national database. One compared board-certified and non-certified surgeons on three types of surgeries.17 Findings revealed fewer deaths when certified surgeons performed peptic ulcer surgery as compared with non-certified surgeons. However, the numbers of deaths related to surgery for stomach cancer and abdominal aneurysm did not differ by certification status. Using the same database, they also compared internal medicine and family practice physicians based on certification status within specialty.18 Outcomes included rates of mortality during cardiac catheterization and in-hospital mortality secondary to a myocardial infarction. Although no significant relationship existed between certification status and mortality during catheterization, board-certified physicians within both specialties had fewer inpatient deaths due to myocardial infarction than did their non-certified colleagues. Pearce and colleagues19 compared board-certified surgeons with subspecialty certification in vascular surgery from the American Board of Surgery with non-certified general surgeons on three procedures: (1) carotid endarterectomy (CEA), (2) lower-extremity bypass graft, and (3) repair of a ruptured abdominal aortic aneurysm (AAA). Patients treated with CEA by board-certified surgeons had a 15% lower risk of death or complication than did patients treated by non-certified surgeons, and a 24% lower risk following treatment for AAA. Certification status did not significantly affect outcomes following lower-extremity bypass grafting. The final study in this cluster, that of Ramsey and colleagues,6 evaluated certified and non-certified internists on a series of outcome variables ranging from evaluations of clinical skills rated by professional colleagues to satisfaction ratings by patients and clinical data abstracted from charts. These authors reported that four of seven results were associated positively with certification status.
The second cluster of papers summarizes studies that grouped physicians from different specialties. Two papers identified positive associations between board certification and outcomes,20,21 while Brook et al.22 reported no association between certification status and complications following CEA. Tussing and Wojtowycz14 found that board-certified obstetricians had a higher cesarean-section rate than did a group of physicians from a mix of specialties. Because the study did not adjust sufficiently for case mix, this finding may be attributable to the fact that the certified obstetricians dealt with more high-risk pregnancies.
The third cluster of papers in Table 1 contains four studies that used information from malpractice claims and medical licensure databases.15,23–25 The 11 results (the Schwartz and Mendelson paper includes four results) demonstrated four positive associations, two negative associations, and five instances with no association. In reviewing professional liability insurance claims in Florida, Sloan and colleagues15 identified negative associations with certification status (i.e., more liability claims among board-certified physicians) for the surgical group of specialties and the combined group of obstetricians—gynecologists with anesthesiologists. The malpractice claims against the medical group of specialists demonstrated no association with certification status. As with the Tussing and Wojtowycz14 study, the results of Sloan's study are difficult to interpret because of inadequate casemix adjustments. The certified physicians could have cared for more complicated patient populations than their non-certified colleagues and generated more malpractice claims. We excluded the Sloan paper from the final tally of findings. In addition, it is acknowledged in these papers using malpractice databases that malpractice claims do not necessarily reflect inferior quality of care.
The general public, health care providers, health care payers, and physicians significantly value specialty board certification. More important, empirical evidence supports the value of board certification. Certification has been associated with increased medical knowledge,6 superior training,8 and certain aspects of patient care.6,19,20 Although these surrogate markers support the value of board certification, they are not direct measures of the clinical care associated with it. In this era of evidence-based medicine, clinical outcomes have become the “gold standard” for evaluating the quality of care. This study represents the first comprehensive review of the literature exploring the relationship between board certification and clinical outcomes. Two conclusions emerge. The first was the surprising finding that only a limited number of published studies have rigorously examined this question. Second, among the reviewed studies, over half the findings support the conclusion that board certification is associated with positive clinical outcomes.
Of the papers meeting the inclusion criteria, no two measured the same outcome variable within the same specialty, and few involved the same specialty. The variability in study design and the range of outcomes measured prevented us from using meta-analytic statistical methods.26 Also, none of the studies reported or adjusted the results to account for the time intervals between board certification, which usually occurs immediately following completion of specialty-specific training, and the dates at which clinical care outcomes were measured.
One might argue that the two studies not adjusting results for case mix should be included in the final analysis.14,15 If these studies are added to the findings, there are 13 papers with 16 positive findings associating certification status and positive outcomes, three negative findings, and 14 with no association. Since adjusting for case mix is a commonly accepted procedure when reporting findings, we have not included these studies.
Other methodologic limitations identified among the studies were of three kinds. The most common was incomplete verification of board certification status. Verifying certification status is critical, because up to 18% of physicians misrepresent their clinical credentials.27,28 Any study exploring associations between board certification and health outcomes must assure all data are valid regardless of whether they represent the board-certification variable or the health-outcomes variables.
A second limitation concerns the methodologic unit of analysis to obtain a stable estimate of each physician's measured patient care outcomes.29 However, most studies pooled patient data across physicians, negating the possibility of measuring an individual physician's performance. An alternative approach would be to implement a nested statistical design grouping each physician's patient data with the patients' outcomes analyzed by physician.30 Patients' data grouped by physician provides a more realistic estimate of each physician's performance, permitting statistical adjustments for unique physician's characteristics. For example, for a study population of 200 non-certified physicians enrolled in a study consisting of 100 physicians with excellent outcomes and many patients and 100 with poor outcomes and few patients, if the data are pooled across all patients, ignoring the physicians' sources, reporting the result for all 200 physicians would misrepresent the underlying reality of patients' outcomes per physician. Only three of the 56 studies meeting the second screening criteria analyzed patients' outcomes by physician using a nested research design.6,16,20
A third methodologic limitation was combining data for physicians from specialties into a single grouping based on certification status. This design significantly limits interpretation of the findings and prohibits comparing outcomes attributed to a single specialty.
A study by Norcini and colleagues31 published after our review represents one of the more methodologically sound designs of board certification and outcomes, although this study pooled results for family practitioners, internists, and cardiologists. The treatment of acute myocardial infarction was compared for certified and non-certified physicians during 1993 using data generated by the Pennsylvania Health Care Cost Containment Council. Patient mortality was used as the outcome measure. After adjusting for hospital resources and other variables, board certification (combined data across all specialties) was associated with a 15% reduction in mortality.
Future research exploring the associatioin between board certification and clinical outcomes is severely limited by the fact that over 87% of licensed physicians in the United States have attained board certification, limiting the pool that can be included in the potential comparison groups of non-certified physicians. Despite the lack of unequivocal evidence documenting the value of board certification, we do not advocate removing it as a measure of expertise. Intuition, expert opinion, surrogate markers, and the findings reported here support the ABMS position that board certification is but one of several important considerations in evaluating a physician's knowledge, skill, and ability to provide good clinical care. In addition to board-certification status, many factors unrelated to the physician affect clinical outcomes, such as the type of clinical setting, size of support staff, and systems of clinical care, to name a few. The conclusions of this review must be considered within the larger context of care—a context of systems as described by the recent Institute of Medicine report on errors in medicine.32
Since board certification is evolving into a virtual expectation for clinical practice in the United States, future research designs may need to group physicians based on numbers of attempts it took to pass the boards and actual board scores, or the amount of time since last taking the certification examination. More recently the ABMS member boards have introduced a recertification program, which requires physicians to revalidate their certifications every six to ten years. Some of the member boards (e.g., the American Boards of Internal Medicine, Family Practice, and Emergency Medicine) currently accumulate data about physicians' performances in practice for recertification. Through the initiatives of the ABMS, the member boards will replace recertification with a Maintenance of Certification that includes a requirement for assessing practice performance.33 Selecting or developing valid outcome measures of practice performance for specialty boards' databases would make it feasible to examine the relationships between board certification and patients' outcomes throughout physicians' careers. Perhaps one lesson to be learned from this review is the need to thoughtfully examine this recertification process to document its value and assure the American public that continued certification is a marker of highquality care.
1. American Medical Association Masterfile and American Board of Medical Specialties' database statistics for 1998.
2. American Board of Medical Specialties. Annual Report and Reference Handbook. Evanston, IL: American Board of Medical Specialties, 2000:98–101.
3. Joint Commission on Accreditation of Healthcare Organizations, 〈http://www.JCAHO.org
〉. Accessed 2/18/00. Joint Commission on Accreditation of Healthcare Organizations, Oakbrook, IL, 2000.
4. Health Plan Employer Data and Information Set, 〈http://www.ncqa.org.htm
〉. Accessed 2/18/00. National Committee for Quality Assurance, Washington, DC, 2000.
5. American Board of Medical Specialties. Annual Report and Reference Handbook. Evanston, IL: American Board of Medical Specialties, 2000:69.
6. Ramsey PG, Carline JD, Inui TS, Larson EB, LoGerfo JP, Wenrich MD. Predictive validity of certification by the American Board of Internal Medicine. Ann Intern Med. 1989;110:719–26.
7. Norcini JJ, Lipner RS, Benson, JA Jr, Webster GD. An analysis of the knowledge base of practicing internists as measured by the 1980 recertification examination. Ann Intern Med. 1985;102:385–9.
8. Norcini JJ, Webster GD, Grosso LJ, Blank LL, Benson JA Jr. Ratings of residents' clinical competence and performance on certification examinations. J Med Educ. 1987;62:457–62.
9. American Board of Medical Specialties. Annual Report and Reference Handbook. Evanston, IL: American Board of Medical Specialties, 2000:68.
10. Cooper H. The Integrative Research Review: A Systematic Approach. Beverly Hills, CA: Sage Publications, 1984.
11. Greenfield S, Rogers W, Mangotich M, Carney MF, Tarlov AR. Outcomes of patients with hypertension and non-insulin-dependent diabetes mellitus treated by different systems and specialties: results from the medical outcomes study. JAMA. 1995;274:1436–44.
12. Ferry ME, Lamy PP, Becker LA. Physicians' knowledge of prescribing for the elderly. A study of primary care physicians in Pennsylvania. J Am Geriatr Soc. 1985;33:616–25.
13. Nelson HS, Areson J, Reisman R. A prospective assessment of the remote practice of allergy: comparison of the diagnosis of allergic disease and the recommendations for allergen immunotherapy by board-certified allergists and a laboratory performing in vitro assays. J Allergy Clin Immunol. 1993;92:380–6.
14. Tussing AD, Wojtowycz MA. The effect of physician characteristics on clinical behavior: cesarean section in New York State. Soc Sci Med. 1993;37:1251–60.
15. Sloan FA, Mergenhagen PM, Burfield WB, Bovbjerg RR, Hassan M. Medical malpractice experience of physicians: predictable or haphazard? JAMA. 1989;262:3291–7.
16. Heck DA, Robinson RL, Partridge CM, Lubitz RM, Freund DA. Patient outcomes after knee replacement. Clin Orthop Rel Res. 1998;356:93–110.
17. Kelly JV, Hellinger FJ. Physician and hospital factors associated with mortality of surgical patients. Med Care. 1986;24:785–800.
18. Kelly JV, Hellinger FJ. Heart disease and hospital deaths: an empirical study. Health Serv Res. 1987;22:369–95.
19. Pearce WH, Parker MA, Feinglass J, Ujiki M, Manhein LM. The importance of surgeon volume and training in outcomes for vascular surgical procedures. J Vasc Surg. 1999;29:768–76.
20. Haas JS, Orav EJ, Goldman L. The relationship between physicians' qualifications and experience and the adequacy of prenatal care and low birthweight. Am J Public Health. 1995;85:1087–91.
21. Nelsen DA, Hartley DA, Christianson J, Moscovice I, Chen MM. The use of new technologies by rural family physicians. J Fam Pract. 1994;38:479–85.
22. Brook RH, Park RE, Chassin MR, Kosecoff J, Keesey J, Solomon DH. Carotid endarterectomy for elderly patients: predicting complications. Ann Intern Med. 1990;113:747–53.
23. Adamson JE. Five-year history of the American Society of Plastic and Reconstructive Surgeons, 1969–1973. Plast Reconstr Surg. 1975;55:445–55.
24. Schwartz WB, Mendelson DN. Physicians who have lost their malpractice insurance: their demographic characteristics and the surplus-lines companies that insure them. JAMA. 1989;262:1335–41.
25. Morrison J, Wickersham P. Physicians disciplined by a state medical board. JAMA. 1998;279:1889–93.
26. Mullen B. Advanced Basic Meta-analysis, Version 1.10. Hillsdale, NJ: L. Erlbaum Associates, 1989.
27. Schaffer WA, Rollo FD, Holt CA. Falsification of clinical credentials by physicians applying for ambulatory-staff privileges. N Engl J Med. 1988;318:356–8.
28. Committee on Government Operations. Patients at Risk: A Study of Deficiencies in the Veterans Administration Medical Quality Assurance Program. Washington, DC: Government Printing Office, 1986.
29. Tamblyn R. Is the public being protected? Prevention of suboptimal medical practice through training programs and credentialing examinations. Eval Health Prof. 1994;17:198–21.
30. Toothaker LE. Multiple Comparisons for Researchers. Newbury Park, CA: Sage Publications, 1991.
31. Norcini JJ, Kimball HR, Lipner RS. Certification and specialization, do they matter in the outcome of acute myocardial infraction? Acad Med. 2000;75:S68–S70.
32. Institute of Medicine. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy of Sciences, 2000.
33. American Board of Medical Specialties. Annual Report and Reference Handbook. Evanston, IL: American Board of Medical Specialties, 2000:76.