Managed care organizations and other health care systems increasingly rely on profiles of physicians to monitor the quality and cost of care. 1–4 Typically, physician profiles are generated from medical claims data and are used to hold physicians accountable for quality of care, use of services, and costs. Profiling, at least at an organizational level, has been recently extended to include assessments in health. 5,6 The Health Care Financing Administration’s Health of Seniors Program plans to use the Medical Outcomes Study 36-Item Short Form Health Survey (SF-36) to assess changes in the physical and mental health status of elderly patients enrolled in different HMOs across the country. 7 Because profiling has important implications for physician credentialing, discipline, and payment, 4,8,9 it is critical that it be appropriately conducted. 10
See p 4
In a previous study, we demonstrated that one measure of patient socioeconomic status (SES), education, had a significant impact on the ranking of physicians according to their patients’ physical and mental health, but not satisfaction, after case-mix adjustment. 11 Patient SES also has appreciable effects on quality measures and inpatient costs beyond those of case-mix adjustment alone. 12,13 Moreover, physician effects account for at most 4% of the variance in diabetic patients’ hospitalizations, visits, laboratory utilization, and glycemic control after adjustment for patient age, gender, case mix, health status, and SES. 14 Given these findings, it is likely that profiles that fail to adjust for patient SES overstate the performance of physicians who care for more affluent patients while penalizing physicians who work with poorer patients.
Managed care organizations and other health organizations cannot easily adjust physician profiles for patient SES because they do not routinely collect socioeconomic data on their patients. One alternative to asking patients about their years of schooling, family income, or occupation is the use of census-based methodology that relies on existing data. 15 Because patient address is correlated with income, education, occupation, and wealth, patients’ zip codes or census blocks serve as indicators of their SES. This methodology has been previously used to examine the relationship between SES and health outcomes within HMOs. 15–18
In this study, we compare the effects on physician profiles of adjustment with 3 different measures of patient SES: one derived by geocoding patient addresses to the census block group, one using patients’ zip codes, and the third based on patient reported education. In all models, SES is assumed to be a patient variable, ie, exogenous to physician performance. Prior research supports this assumption. 11,19 That is, there is no evidence that physicians caring for poorer patients or those with less education are worse doctors. In this study, comparisons are based primarily on changes in physician rank associated with adjusting or not adjusting for the SES variables. It is assumed that if physicians’ ranks for a given performance indicator change after adjustment for the SES variable (a patient-related factor), then that revised ranking reflects a better measure of physician performance, one that more properly adjusts for a patient-related effect.
Physician and Patient Samples
We analyzed data from a study of primary care physician referral behavior. 11,20 As part of that study, 100 primary care physicians (internists and family physicians) in the metropolitan Rochester, New York, area were randomly selected for participation on the basis of the following eligibility criteria: (1) having been in practice for ≥2 years before the office survey; (2) having ≥100 patients enrolled in both of those years in the largest local managed care organization (which enrolls >50% of the local population); and (3) not having an area of specialization that resulted in referrals from other physicians (assessed by physician self-report).
Self-administered surveys were distributed by research assistants to ≥50 consecutive patients ≥30 years of age visiting each of the offices of the participating physicians. The research assistants assisted patients who needed help completing the questionnaire. In addition to completing health status, patient satisfaction, and morbidity instruments, patients were asked to provide their age, gender, education, time since their previous visit to the physician, and address.
Physician Practice Profiles
Physicians were ranked on the basis of their mean patients’ satisfaction, physical health status, and mental health status derived from patient surveys conducted in the physicians’ offices. All rankings were adjusted for patient age, gender, and insurance status.
Patient Selection and Survey.
Self-administered surveys were distributed by research assistants to ≥50 consecutive patients ≥30 years of age visiting each of the offices of the participating physicians. The research assistants obtained health insurance information on each patient and guided patients in completing the questionnaire. In addition to completing health status, and morbidity instruments, patients were asked to provide their age, gender, address, years of completed education, and time since previous visit.
Patient Health Status.
Patient physical and mental health status were assessed using the 12-item Medical Outcomes Study SF-12, a self-report instrument that allows construction of a physical health component summary (PCS-12) and a mental health component summary (MCS-12). 21 Previous studies have confirmed the reliability and validity of the SF-12. 21–24
Patients completed the general satisfaction subscale from the Patient Satisfaction Questionnaire. This measure has been found to be reliable and valid. 25
Patient morbidity was assessed from a checklist of chronic diseases and treatments for those diseases based on the method described by Katz et al. 26
Self reported years of education was used as an individual-level measure of SES.
The patient’s insurance status was recorded by the research assistant using billing information and coded into 1 of the following categories: private HMO, private non-HMO, Medicaid non-HMO, Medicaid HMO, Medicare, Workers Compensation, and other.
Census-Based Socioeconomic Indicators.
Patient addresses were geocoded to the block group level with Atlas GIS (Claritas Inc) and US Census Bureau TIGER/Line 1998 files. Information from the 1990 census was used to derive indicators of SES at the census block group and zip code levels. The indicators used were percent of residents with ≥12 years of education, median per capita income, percent of residents in blue collar occupations, and percent of African Americans. The values of the census-based socioeconomic indicators were appended to each observation for each patient.
Three sets of mixed-effects linear regression analyses were fit, 27 one for each performance outcome: satisfaction and physical and mental health status. Mixed-effects models adjust for the clustering induced by the nesting of patients within practices. They also produce better estimates of relative physician performance (profiling). 28–30 The models, also called empirical Bayes models, estimate the variation in patient outcomes by physician (a random effect in the model) after adjusting for patient characteristics (fixed effects). Patient factors included in all models were age, age squared, gender, and insurance status (together called demographic variables). Sets of additional patient variables included in selected models were clinical conditions (case mix), patient education, census-based SES derived at the block group level, and census-based SES derived at the zip code level. To avoid overrepresentation by frequently visiting patients, the analyses were weighted by the interval since the patient’s previous visit to the physician.
Rankings of physicians for each outcome were determined from the parameter estimates for the physician random effects. By including or excluding each set of additional patient variables from the model and then examining the absolute changes in ranks produced (in either direction), we examined the impact on observed physician performance of adjustment for that set of additional patient variables. To further assess the impact of each set of additional patient variables, we report (1) the Spearman rank correlation between the ranking with the additional patient variable included with the ranking with that variable excluded, (2) the reduction in the physician variance component produced by the additional patient variable, and (3) the change in Akaike’s information criterion produced by the additional patient variable (a measure of improvement in model fit). All analyses were conducted with the MIXED procedure in SAS version 6.12 (SAS Institute).
Of the 5,055 patients who participated in the study, 4,949 (98%) provided addresses, and 4,025 (81%) were successfully geocoded to the block group level. The mean patient education level was 13.4 years (SD, 2.4 years). The mean education level of patients whose addresses could not be successfully geocoded was not statistically significantly different from those successfully matched (0.13 years higher; 95% CI, −0.04–0.29). The mean physical component summary score was 44.5 (SD, 11.5), mean mental component summary score was 49.1 (SD, 10.7), and mean satisfaction score was 4.02 (SD, 0.39); all these variables were approximately normally distributed. The mean practice (physician) -level descriptors averaged across the ∼50 patients sampled in each practice have been reported previously. 11
The changes in physician ranks for patient satisfaction and physical and mental health after adjustment for selected additional variables are shown in Tables 1 through 3. Also shown is the change in physician variance component and Akaike’s information criterion. When the mean ranking change is close to 0, the Spearman rank correlation is close to 1, and the change in physician variance component and Akaike’s information criterion are close to 0, then the effects of the additional adjustment are negligible.
As shown in Table 1, adjustment for patient zip code SES results in greater change in physician rank for patient satisfaction than is observed with the other 2 SES measures. The other SES indicators suggest relatively little impact on the physician rankings. When zip code is added to either of the other 2 SES measures, the effect is comparable to that observed when zip code is added to demographic factors alone or to demographic and case-mix factors.
The effects of the SES measures on physician rankings for physical health are shown in Table 2. The effects of the 3 SES measures on rankings are similar, ranging from 5.3 mean change in ranking when block group SES is added to 4.5 when education is added. The changes in ranking are more modest when education is added to block group or zip code or the converse and are most modest when zip code SES is added to block group SES or the converse. Similar effects were observed for rankings of mental health status (Table 3), but the effects are more modest.
For comparison of effect size, it was found that the addition of case mix to demographics produced a mean physician rank change of 1.6, 10.3, and 9.3 for patient satisfaction, physical health status, and mental health status, respectively.
Having previously shown that adjustment for patient education affects physician profiles for patient physical and mental health, 11 we examined the effects using SES measures based on patient addresses geocoded to the census block group and zip code levels. Overall, we observed similar effects on case-mix–adjusted profiles using the census block group measure, the zip code measure, and patient education. However, there were differences in the relative effects of the 3 SES measures on the rankings for patient satisfaction and physical and mental health status. The effects on physician rankings for patient satisfaction using the zip code SES were greater than those of the other 2 measures, which had negligible effects. For the physical and mental health status rankings, the 2 census-derived measures had similar effects. Individually measured education did not yield a greater effect on physician ranking than either of the census-derived measures for any of the patient outcomes. These findings were observed despite the use of 7-year-old census data. Previous studies show that the age of the census data has relatively little impact on SES measurement error. 31 Nonmatching of addresses did not appear to be associated with bias in measurement. That is, those successfully geocoded had education levels very similar to those not matched.
These findings, if replicated, suggest that patient zip code–level SES may offer a convenient alternative to individually collected SES data for adjusting physician profiles. Use of patient zip codes offers the additional advantage over block group–derived SES of omitting the need for geocoding addresses and results in less loss of data resulting from unsuccessful matching. Zip codes are based on larger areas than block groups, typically containing ∼25,000 residents compared with ∼1,000 in block groups. Consequently, they are likely to contain considerably more socioeconomic heterogeneity than do block groups. Previous work tends to show stronger effects when smaller units of aggregation are used. 15,32,33 Furthermore, census-level variables, while easier to collect, only partially capture attributes of the individual. 15,17,31 Thus, the results of this study are somewhat surprising.
Both census-derived measures capture components of SES beyond education, such as income, occupation, and race/ethnicity. The census-derived measures also capture contextual effects of the environment in which the person lives. 16,34–36 It is possible that the additional socioeconomic and contextual information captured in the census-derived measures offsets the loss of individual-level information. Also, the goal in physician profiling is to measure average patient socioeconomic effects. At this aggregated level, the loss of precision of individually gathered information is less important. The relatively superior performance of the zip code measure is harder to account for. It is possible that contextual effects may be more important, or more reliably accounted for, at the larger area encompassed by the zip code than the relatively smaller block group area.
As previously noted, 11 our data suggest that profiles not accounting for differences in patient SES by physician practice may underestimate the performance of physicians caring for poorer patients while overestimating the performance of those with more affluent practices. This socioeconomic bias in profiling may encourage selective enrollment of more affluent patients in an effort to improve performance ratings. Our findings suggest that the use of geocoding or patient zip codes may mitigate this bias.
These findings should be tempered by the limitations of the data. The findings are based on a single sample of physicians from a single community and may not necessarily generalize to other communities. Greater patient mobility over time, more socioeconomic heterogeneity by census block group, or less variability in patient SES between and within physician practices might produce less dramatic effects. Furthermore, we were not able assess the statistical significance of differences in the changes in rank observed between the different models because of the absence of a satisfactory statistical test. Thus, we cannot quantify the probability of chance accounting for our findings. However, the consistency of the findings across the 3 domains militates against this explanation. Last, we examined the impact only on 3 measures of physician performance. We did not examine effects on utilization, costs, or clinical preventive services compliance. However, at least 1 previous study suggests that adjustment with census-based measures affects HEDIS screening and preventive services. 12
In conclusion, census-derived measures of SES, particularly those based on zip code data, may offer a convenient and relatively inexpensive way for health care organizations to improve the quality of physician profiling. Further study is needed to confirm these findings and delineate more fully the limitations of this approach.
1. Spoeri RK, Ullman R. Measuring and reporting managed care performance: Lessons learned and new initiatives. Ann Intern Med 1997; 127: 726–732.
2. Epstein AM. Rolling down the runway: The challenges ahead for quality report cards. JAMA 1998; 279: 1691–1696.
3. Brook RH, McGlynn EA, Cleary PD. Quality of health care, part 2: Measuring quality of care. N Engl J Med 1996; 335: 966–970.
4. McNeil BJ, Pedersen SH, Gatsonis C. Current issues in profiling quality of care. Inquiry 1992; 29: 298–307.
5. Lohr KN. Applications of health status assessment measures in clinical practice: Overview of the Third Conference on Advances in Health Status Assessment. Med Care 1992; 30: MS1–MS14.
6. Patrick DL. Finding health-related quality of life outcomes sensitive to health-care organization and delivery. Med Care 1997; 35: NS49–NS57.
7. Cooper JK. Measuring the health of seniors. Arch Fam Med 1998; 7: 415–416.
8. Sessa EJ. Information is power: The Pennsylvania experiment. J Health Care Benefits 1992; Jan-Feb: 44–48.
9. Blum JD. The evolution of physician credentialing into managed care selective contracting. Am J Law Med 1996; 22: 173–203.
10. Kassirer JP. The use and abuse of practice profiles. N Engl J Med 1994; 330: 634–636.
11. Fiscella K, Franks P. Influence of patient education on profiles of physician practices. Ann Intern Med 1999; 131: 745–751.
12. Zaslavsky AM, Hochheimer JN, Schneider EC, et al. Impact of sociodemographic case-mix on the HEDIS measures of health plan quality. In: Proceedings from the AHSR 16th Annual Meeting, June 28, 1999; Chicago, Ill; 1999:327–328. Abstract.
13. Pickens G, Bankowitz R, Bouman P. Can we measure the impact of social and economic status on risk-adjusted costs of inpatient care? In: Proceedings from the AHSR 16th Annual Meeting, June 28, 1999; Chicago, Ill; 1999:388–389. Abstract.
14. Hofer TP, Hayward RA, Greenfield S, et al. The unreliability of individual physician “report cards” for assessing the costs and quality of care of a chronic disease. JAMA 1999; 281: 2098–2105.
15. Krieger N. Overcoming the absence of socioeconomic data in medical records: Validation and application of a census-based methodology. Am J Public Health 1992; 82: 703–710.
16. Krieger N. Women and social class: A methodological study comparing individual, household, and census measures as predictors of black/white differences in reproductive history. J Epidemiol Community Health 1991; 45: 35–42.
17. Cherkin DC, Grothaus L, Wagner EH. Is magnitude of co-payment effect related to income? Using census data for health services research. Soc Sci Med 1992; 34: 33–41.
18. Selby JV, Fireman BH, Swain BE. Effect of a copayment on use of the emergency department in a health maintenance organization. N Engl J Med 1996; 334: 635–641.
19. Starfield B, Powe NR, Weiner JR, et al. Costs vs quality in different types of primary care settings. JAMA 1994; 272: 1903–1908.
20. Franks P, Zwanziger J, Mooney C, et al. Variations in primary care physician referral rates. Health Serv Res 1999; 34 (pt 2):323–329.
21. Ware JE Jr, Kosinski M, Keller SD. A 12-item short-form health survey: Construction of scales and preliminary tests of reliability and validity. Med Care 1996; 34: 220–233.
22. Jenkinson C, Layte R, Jenkinson D, et al. A shorter form health survey: Can the SF-12 replicate results from the SF-36 in longitudinal studies? J Public Health Med 1997; 19: 179–186.
23. Johnson JA, Coons SJ. Comparison of the EQ-5D and SF-12 in an adult US sample. Qual Life Res 1998; 7: 155–166.
24. Gandek B, Ware JE, Aaronson NK, et al. Cross-validation of item selection and scoring for the SF-12 Health Survey in nine countries: Results from the IQOLA Project, International Quality of Life Assessment. J Clin Epidemiol 1998; 51: 1171–1178.
25. Ware JE, Snyder MK, Wright WR, et al. Defining and measuring patient satisfaction with medical care. Eval Program Planning 1984; 6: 247–263.
26. Katz JN, Chang LC, Sangha O, et al. Can comorbidity be measured by questionnaire rather than medical record review? Med Care 1996; 34: 73–84.
27. Zeger SL, Liang KY. An overview of methods for the analysis of longitudinal data. Stat Med 1992; 1: 1825–1839.
28. Christiansen CL, Morris CN. Improving the statistical approach to health care provider profiling. Ann Intern Med 1997; 127: 764–768.
29. Normand ST, Glickman ME, Gatsonis CA. Statistical methods for profiling providers of medical care: Issues and applications. J Am Stat Assoc 1997; 92: 803–814.
30. Goldstein H, Spiegelhalter DJ. League tables and their limitations: Statistical issues in comparisons of institutional performance. J Royal Stat Soc 1996; 159: 385–443.
31. Geronimus AT, Bound J. Use of census-based aggregate variables to proxy for socioeconomic group: Evidence from national samples. Am J Epidemiol 1998; 148: 475–486.
32. Hyndman JC, Holman CD, Hockey RL, et al. Misclassification of social disadvantage based on geographical areas: Comparison of postcode and collector’s district analyses. Int J Epidemiol 1995; 24: 165–176.
33. Davey SG, Hart C. Re: “Use of census-based aggregate variables to proxy for socioeconomic group: Evidence from national samples.” Am J Epidemiol 1999; 150: 996–997.
34. Duncan C, Jones K, Moon G. Context, composition and heterogeneity: Using multilevel models in health research. Soc Sci Med 1998; 46: 97–117.
35. Brooks-Gunn J, Duncan GJ, Klebanov PK, et al. Do neighborhoods influence child and adolescent development? Am J Sociol 1993; 99: 353–395.
36. Robert SA. Community-level socioeconomic status effects on adult health. J Health Soc Behav 1998; 39: 18–37.