Virtually all hospitals in the rankings are teaching hospitals. One likely reason is the inclusion criteria, which specify membership in the Council of Teaching Hospitals or the presence of cutting-edge technology; both of these criteria favor teaching hospitals.13 In addition, the physician survey that produces the reputation score asks respondents to “list the five hospitals (and/or affiliated medical schools) [italics added] in the United States that you believe provide the best care for patients with the most serious and difficult medical problems.” This may suggest to respondents that a medical school affiliation is desirable. Studies exploring whether teaching hospitals have better outcomes than their nonteaching counterparts have shown lower mortality in teaching hospitals, but some also found higher costs,30–33 and in one study teaching hospitals performed less well in the areas of prevention and health counseling than did nonteaching hospitals.34 Two other criteria in the rankings—third-party recognition and designation35–38 and higher nurse staffing ratios39,40—also were associated with lower patient mortality after adjusting for patient and other characteristics.
One systematic study has shown that only 55% of care delivered across all hospitals conforms to accepted best practice,41 and recent systems for assessing hospital quality take into consideration adherence to accepted clinical guidelines that experts have declared important in avoiding unnecessary variation and in increasing the quality and efficiency of care.42,43 Two studies reported that the improved outcomes in ranked hospitals may relate to greater adherence to treatment guidelines.17,18 However, a study comparing quality rankings based on adherence to clinical guidelines (the Medicare program’s Hospital Compare system) found that fewer than 50% of the hospitals in the “America’s Best Hospitals” rank list for cardiac diagnoses and only 15% of hospitals ranked for respiratory disorders also had top quartile Hospital Compare scores for these diagnoses. Further, only one third of “America’s Best Hospitals” Honor Roll institutions placed in the top quartile of the Hospital Compare system.44 The University Healthsystem Consortium (UHC) included two Honor Roll institutions in its ranking of the five top-performing hospitals.45 The other institutions in the UHC ranking were not represented in the “America’s Best Hospitals” institutional Honor Roll and were represented only to a limited degree in the individual specialties rankings.
To explore the link between clinical and educational quality, the analysis assessed the performance of ranked programs on accepted measures of educational quality, including accreditation performance, NRMP performance, and, to the extent available, graduates’ board certification performance. For accreditation performance, residency programs in the ACGME’s database of accredited programs were matched to the “America’s Best Hospitals” rankings, producing a sample of 390 accredited programs in institutions that were ranked in the clinical specialty represented by the program. Analysis of their accreditation performance showed a much lower rate of adverse actions (less than 1% compared with an adverse action rate of 8% for all ACGME-accredited programs). In addition, the interval between accreditation review, which reflects ACGME’s and the review committee’s comfort with compliance and educational quality, is significantly longer for ranked programs (P = .02). Analysis of ranked programs’ performance in the 2006 and 2007 resident matches showed all had high fill rates, but it was not possible to distinguish their performance from that of a sizable share of nonranked programs. In addition, match performance reflects programs’ hiring success, not educational achievements.
American Board of Medical Specialties (ABMS) certification examination pass rates offer valid information about educational outcomes, albeit in a limited range emphasizing medical knowledge. To date, only two ABMS member boards (the American Board of Internal Medicine and the American Board of Pediatrics) have released pass rate data for individual programs. Pediatrics is the only specialty in the “America’s Best Hospitals” rankings with this information available. Analysis showed that pediatrics programs in ranked institutions outperformed the national average, with an ABMS certification pass rate of 93.8% compared with a national average of 78.9%; ranked programs’ percentage of graduates taking the exam also is slightly higher (94.9% versus 91.1%). To assess this relationship for 25 pediatrics programs represented in the rankings in more detail, the analysis explored the relationship between programs’ place in the 2006 rankings and their rank by 2005–2006 ABMS certification pass rate. The analysis used the Kruskal-Wallis test, a nonparametric statistic for examining groups, in which the actual observations in a group are replaced by their rank.46 The value of 0.422 on the Kruskal-Wallis test suggests a small positive relationship between the two rankings, despite the range restriction in the data.
Clinical “centers of excellence” (in which a group of highly trained and specialized experts focus on a particular clinical area) for cancer and cardiac care and digestive disorders found in ranked hospitals may be of potential importance for the learning environment they create. The high degree of specialization of physicians likely found in these centers has been shown to improve performance in diagnosis and care,47 though this advantage seems to be limited to complex cases.48 Reasons for this limitation include that experts care for patients with similar diseases, work in settings with better equipment, and have more knowledgeable colleagues, all of which increase performance feedback.49 These factors also could contribute to a better learning environment in ranked institutions. If ranked institutions have better adherence to guidelines that promote efficient, high-quality care, this may be relevant to educational outcomes, because residents seem to adopt the practice attributes of their educational setting, as evidenced by the relatively lower performance of residents training in an environment with few constraints on exams about appropriate use of medical resources.50
The analyses showed that institutions included in the “America’s Best Hospitals” rankings outperformed comparison institutions, with improved performance in clinical dimensions and in the limited reported attributes linked to the learning environment for resident physicians. However, the rankings have three limitations that may reduce their utility for making judgments about clinical or educational quality. First, ranked institutions’ greater renown may allow them to be more selective in recruiting residents, with better educational outcomes resulting from this, not a superior learning environment provided by the residency programs in these institutions. Second, peer judgments about quality used to determine reputation scores may be based on the prestige of the institutions in which the services reside, suggesting that the specialty rankings and their reaggregation into an institutional Honor Roll may be confounded. Finally, the rankings are limited to specialties that emphasize specialized services over prevention and health maintenance, and the value placed on cutting-edge technology may persuade hospitals to invest in such innovations, contributing to overuse and added burden on the health care system.
One limitation of this systematic review of the data underlying the rankings is the lack of primary studies in many areas relating to measures of quality underlying the standards. The available literature allows only a tantalizing glimpse at the relationship between improved clinical and educational outcomes, and it does not truly answer the question of how they coexist and interact in teaching institutions. Research is needed to more fully explore the attributes of quality pertaining to the structure and process of care and how this affects outcomes. To enhance the usefulness of the rankings in assessing clinical and educational excellence, the rankings would need to incorporate added valid quality indicators. Possible quality indicators for which some information is presently available include factors affecting mortality, such as medical errors,51,52 and information on efforts to design safe systems of care.53–55 From an educational perspective, the rankings do not include information on team work, reflective practice, or efforts to use data to improve clinical quality—concepts considered vital to the education of physicians and other health professionals.56,57
As they are currently constituted and used, the rankings permitted only a tantalizing glimpse at the relationship between clinical and educational quality, and they do not answer the question of how the two coexist and interact in teaching settings. The rankings’ most serious limitation for the purpose of improving clinical and educational quality in the United States is that the number of ranked programs and institutions is too small to affect care or the professional development of physicians—unless these institutions perform an added role as sites for the development, study, and dissemination of best practices. That this may occur already is shown in articles on quality improvements originating from a several Honor Roll institutions relating to clinical performance58–61 and enhancing the learning environment for physicians and other health professionals.62–64 Expanding this work to a larger number of institutions and to the education of health professionals is likely to have a more profound effect on health care quality for Americans than would efforts to develop the ideal quality ranking.
It is encouraging for programs and institutions wishing to improve their performance that the dearth of comparative information need not be an impediment to using available data to improve clinical care and education, either independently or in collaboration with partners. Going beyond these efforts to a national dataset on clinical and educational quality will require research to refine measures of educational achievement and performance in practice to expand the small body of evidence that institutions with a better care environment seem to produce graduates superior in measures of competence for practice. This would also realize the aim of the Outcome Project of finding ways to effectively link quality in resident education and practice.
1 Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus S. General competencies and accreditation in graduate medical education. Health Aff (Millwood). 2002;21:103–111.
4 Mitchell M, Srinivasan M, West DC, et al. Factors affecting resident performance: Development of a theoretical model and a focused literature review. Acad Med. 2005;80:376–389.
5 Borowitz SM, Saulsbury FT, Wilson WG. Information collected during the residency match process does not predict clinical performance. Arch Pediatr Adolesc Med. 2000;154:256–260.
6 Boyse TD, Patterson SK, Cohan RH, et al. Does medical school performance predict radiology resident performance? Acad Radiol. 2002;9:437–445.
7 Clarke M. Quantifying quality: What can the U.S. News and World Report rankings tell us about the quality of higher education? Educ Policy Anal Arch. March 2002;10(16). Available at: (http://epaa.asu.edu/epaa/v10n16
). Accessed October 7, 2008.
8 Chen Y, Xie J. Third-party product review and firm marketing strategy. Mark Sci. 2005;24:218–240.
9 Green J, Wintfeld N, Krasner M, Wells C. In search of America’s best hospitals. The promise and reality of quality assessment. JAMA. 1997;277:1152–1155.
10 McGaghie WC, Thompson JA. America’s best medical schools: A critique of the U.S. News & World Report rankings. Acad Med. 2001;76:985–992.
11 Schneider EC, Epstein AM. Use of public performance reports. JAMA. 1998;279:1638–1642.
12 Hannan EL, Stone CC, Biddle TL, DeBuono BA. Public release of cardiac surgery outcomes data in New York: What do New York state cardiologists think of it? Am Heart J. 1997;134:1120–1128.
13 McFarlane E, Murphy J, Olmsted MG, Drozd EM, Hill C. America’s Best Hospitals. 2007 Methodology. Research Triangle Park, NC: RTI International; July 2007.
14 Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund Q. 1966;44(suppl):166–206.
16 Wang OJ, Wang Y, Lichtman JH, et al. “America’s Best Hospitals” in the treatment of acute myocardial infarction. Arch Intern Med. 2007;167:1345–1351.
17 Chen J, Radford MJ, Wang Y, Marciniak TA, Krumholz HM. Do “America’s Best Hospitals” perform better for acute myocardial infarction? N Engl J Med. 1999;340:286–292.
18 Williams SC, Koss RG, Morton DJ, Loeb JM. Performance of top-ranked heart care hospitals on evidence-based process measures. Circulation. 2006;114:558–564.
19 Flood AB, Scott WR, Ewy W. Does practice make perfect? Part I: The relation between hospital volume and outcomes for selected diagnostic categories. Med Care. 1984;22:98–114.
20 Birkmeyer JD, Siewers AE, Finlayson EV, et al. Hospital volume and surgical mortality in the United States. N Engl J Med. 2002;346:1128–1137.
21 Allareddy V, Allareddy V, Konety BR. Specificity of procedure volume and in-hospital mortality association. Ann Surg. 2007;246:135–139.
22 Barker FG 2nd, Amin-Hanjani S, Butler WE, Ogilvy CS, Carter BS. In-hospital mortality and morbidity after surgical treatment of unruptured intracranial aneurysms in the United States, 1996–2000: The effect of hospital and surgeon volume. Neurosurgery. 2003;52:995–1007.
23 Shackley P, Slack R, Booth A, Michaels J. Is there a positive volume–outcome relationship in peripheral vascular surgery? Results of a systematic review. Eur J Vasc Endovasc Surg. 2000;20:326–335.
24 Halm EA, Lee C, Chassin MR. Is volume related to outcome in health care? A systematic review and methodologic critique of the literature. Ann Intern Med. 2002;137:511–520.
25 Peterson ED, Coombs LP, DeLong ER, Haan CK, Ferguson TB. Procedural volume as a marker of quality for CABG surgery. JAMA. 2004;291:195–201.
26 Epstein AJ, Rathore SS, Krumholz HM, Volpp KG. Volume-based referral for cardiovascular procedures in the United States: A cross-sectional regression analysis. BMC Health Serv Res. 2005;5:42.
27 Rogowski JA, Horbar JD, Staiger DO, et al. Indirect vs. direct hospital quality indicators for very low-birth-weight infants. JAMA. 2004;291:202–209.
28 Begg CB, Riedel ER, Bach PB, et al. Variations in morbidity after radical prostatectomy. N Engl J Med. 2002;346:1138–1144.
29 Schrag D, Panageas KS, Riedel E, et al. Hospital and surgeon procedure volume as proctors of outcome following rectal cancer resection. Ann Surg. 2002;236:583–592.
30 Taylor DH Jr, Whellan DJ, Sloan FA. Effects of admission to a teaching hospital on the cost and quality of care for Medicare beneficiaries. N Engl J Med. 1999;340:293–299.
31 Rosenthal GE, Harper DL, Quinn LM, Cooper GS. Severity-adjusted mortality and length of stay in teaching and nonteaching hospitals. Results of a regional study. JAMA. 1997;278:485–490.
32 Polanczyk CA, Lane A, Coburn M, et al. Hospital outcomes in major teaching, minor teaching, and nonteaching hospitals in New York State. Am J Med. 2002;112:255–261.
33 Zimmerman JE, Shortell SM, Knaus WA, et al. Value and cost of teaching hospitals: A prospective, multicenter, inception cohort study. Crit Care Med. 1993;21:1432–1442.
34 Landon BE, Normand SL, Lessler A, et al. Quality of care for the treatment of acute medical conditions in US hospitals. Arch Intern Med. 2006;166:2511–2517.
35 Sampalis JS, Lavoie A, Boukas S, et al. Trauma center designation: Initial impact on trauma-related mortality. J Trauma. 1995;39:232–237.
36 Aiken LH, Smith HL, Lake ET. Lower Medicare mortality among a set of hospitals known for good nursing care. Med Care. 1994;32:771–787.
37 Birkmeyer NJ, Goodney PP, Stukel TA, Hillner BE, Birkmeyer JD. Do cancer centers designated by the National Cancer Institute have better surgical outcomes? Cancer. 2005;103:435–441.
38 Demetriades D, Martin M, Salim A, et al. Relationship between American College of Surgeons trauma center designation and mortality in patients with severe trauma (injury severity score >15). J Am Coll Surg. 2006;202:212–215.
39 Kane RL, Shamliyan TA, Mueller C, Duval S, Wilt TJ. The association of registered nurse staffing levels and patient outcomes: Systematic review and meta-analysis. Med Care. 2007;45:1195–1204.
40 Needleman J, Buerhaus P, Mattke S, Stewart M, Zelevinsky K. Nurse-staffing levels and quality of care in hospitals. N Engl J Med. 2002;346:1415–1422.
41 Asch SM, Kerr EA, Keesey J, et al. Who is at greatest risk for receiving poor-quality health care? N Engl J Med. 2006;354:1147–1156.
42 Atkins D, Fink K, Slutsky J; Agency for Healthcare Research and Quality; North American Evidence-based Practice Centers. Better information for better health care: The Evidence-based Practice Center program and the Agency for Healthcare Research and Quality. Ann Intern Med. 2005;142:1035–1041.
43 Lohr KN. Rating the strength of scientific evidence: Relevance for quality improvement programs. Int J Qual Health Care. 2004;16: 9–18.
44 Halasyamani LK, Davis MM. Conflicting measures of hospital quality: Ratings from “Hospital Compare” versus “Best Hospitals.” J Hosp Med. 2007;2:128–134.
46 Sheskin DJ. Handbook of Parametric and Nonparametric Statistical Procedures. 3rd ed. Boca Raton, Fla: Chapman & Hall/CRC: 2004.
47 Esserman L, Cowley H, Eberle C, et al. Improving the accuracy of mammography: Volume and outcome relationships. J Natl Cancer Inst. 2002;94:369–375.
48 Simpson SA, Gilhooly KJ. Diagnostic thinking processes: Evidence from a constructive interaction study of electrocardiogram (ECG) interpretation. Appl Cogn Psychol. 1997;11:543–554.
49 Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;79(10 suppl):S70–S81.
50 Fisher ES. The paradox of plenty: Implications for performance measurement and pay for performance. Manag Care. 2006;15(10 suppl 8):3–8.
51 Singh H, Thomas EJ, Petersen LA, Studdert DM. Medical errors involving trainees: A study of closed malpractice claims from 5 insurers. Arch Intern Med. 2007;167:2030–2036.
52 Singh H, Naik AD, Rao R, Petersen LA. Reducing diagnostic errors through effective communication: Harnessing the power of information technology. J Gen Intern Med. 2008;23:489–494.
53 Glance LG, Osler TM, Mukamel DB, Dick AW. Effect of complications on mortality after coronary artery bypass grafting surgery: Evidence from New York State. J Thorac Cardiovasc Surg. 2007;134:53–58.
54 Silber JH, Rosenbaum PR, Williams SV, Ross RN, Schwartz JS. The relationship between choice of outcome measure and hospital rank in general surgical procedures: Implications for quality assessment. Int J Qual Health Care. 1997;9:193–200.
55 Karsh BT, Holden RJ, Alper SJ, Or CK. A human factors engineering paradigm for patient safety: Designing to support the performance of the healthcare professional. Qual Saf Health Care. 2006;15(suppl 1):i59–i65.
56 Davies HTO, Nutley SM, Mannion R. Organisational quality culture and quality of health care. Qual Health Care. 2000;9:111–119.
57 Galbraith RM, Holtman MC, Clyman SG. Use of assessment to reinforce patient safety as a habit. Qual Saf Health Care. 2006; 15(suppl 1):i30–i33.
58 Berenholtz SM, Pronovost PJ, Lipsett PA, et al. Eliminating catheter-related bloodstream infections in the intensive care unit. Crit Care Med. 2004;32:2014–2020.
59 Johnson T, Currie G, Keill P, et al. NewYork-Presbyterian Hospital: Translating innovation into practice. Jt Comm J Qual Patient Saf. 2005;31:554–560.
60 Pronovost PJ, Berenholtz SM, Goeschel CA, et al. Creating high reliability in health care organizations. Health Serv Res. 2006;41: 1599–1617.
61 Pronovost PJ, King J, Holzmueller CG, et al. A web-based tool for the Comprehensive Unit-based Safety Program (CUSP). Jt Comm J Qual Patient Saf. 2006;32:119–129.
62 Lypson ML, Frohna JG, Gruppen LD, Woolliscroft JO. Assessing residents’ competencies at baseline: Identifying the gaps. Acad Med. 2004;79:564–570.
63 Gordon JA, Wilkerson WM, Shaffer DW, Armstrong EG. “Practicing” medicine without risk: Students’ and educators’ responses to high-fidelity patient simulation. Acad Med. 2001;76:469–472.
64 Afessa B, Kennedy CC, Klarich KW, et al. Introduction of a 14-hour work shift model for housestaff in the medical ICU. Chest. 2005;128:3910–3915.