Purpose: The annual U.S. News & World Report (USN&WR) Primary Care Medical School (PCMS) ranking attracts considerable attention, but its measurement properties have not been published. The authors examined the short-term stability of the PCMS ranking and the PCMS score from which it derives, along with the short-term spread of schools’ rankings.
Method: The authors employed published data and methods to reconstruct the 2009–2012 PCMS scores and rankings. They used mixed-effects models to assess the within-school, between-year reliability (short-term stability) of the PCMS score and ranking, yielding intraclass correlation coefficients (ICCs). They defined short-term spread as the median within-school range in ranking across the four-year study period.
Results: Reconstructed PCMS scores correlated highly with published scores all four years (Pearson correlations ≥ 98.9%). Most schools’ mean annual PCMS scores were tightly clustered near the center of the score distribution. ICCs for the PCMS score and ranking were, respectively, 94% and 90%. The median difference between the best and worst ranking over the study period was 4 for the 18 schools with an average annual ranking of 1 to 20, and 17 for the other 89 schools (P < .001, Kruskal-Wallis test).
Conclusions: The short-term stability of the USN&WR PCMS score and ranking were reasonably good. However, the short-term spread in PCMS rankings was large, particularly among schools with mean annual rankings below the top 20. The variability is greater than could be plausibly attributed to actual changes in training quality. These findings raise questions regarding the ranking’s validity and usefulness.
Dr. Tancredi is assistant professor, Department of Pediatrics, Center for Healthcare Policy and Research, University of California Davis School of Medicine, Sacramento, California.
Dr. Bertakis is professor and chair, Department of Family and Community Medicine, Center for Healthcare Policy and Research, University of California Davis School of Medicine, Sacramento, California.
Dr. Jerant is professor, Department of Family and Community Medicine, Center for Healthcare Policy and Research, University of California Davis School of Medicine, Sacramento, California.
Editor’s Note: A commentary by D.G. Kirch and J.E. Prescott appears on page 1064.
Funding/Support: A University of California Davis Department of Family and Community Medicine Research Grant funded, in part, the research presented in this report.
Other disclosures: None.
Ethical approval: The study was exempted under University of California Davis institutional review board policies.
Correspondence should be addressed to Dr. Jerant, Department of Family and Community Medicine, University of California Davis School of Medicine, 4860 Y St., Suite 2300, Sacramento, CA 95817; telephone: (916) 734-7081; fax: (916) 734-5641; e-mail: email@example.com.