Validation of Reported Predialysis Nephrology Care of Older Patients Initiating Dialysis

Abstract

The Centers for Medicare and Medicaid Services (CMS) Medical Evidence Report (form CMS-2728) queries providers about the timing of the patient’s first nephrologist consultation before initiation of dialysis. The monitoring of disease-specific goals in the Healthy People 2020 initiative will use information from this question, but the accuracy of the reported information is unknown. We defined a cohort of 80,509 patients aged ≥67 years who initiated dialysis between July 2005 and December 2008 with ≥2 years of uninterrupted Medicare coverage as their primary payer. The primary referent, determined from claims data, was the first observed outpatient nephrologist consultation; secondary analyses used the earliest nephrology consultation, whether inpatient or outpatient. We used linear regression models to assess the associations among the magnitude of discrepant reporting and patient characteristics and we tested for any temporal trends. When using the earliest recorded outpatient nephrology encounter, agreement between the two sources of ascertainment was 48.2%, and the κ statistic was 0.29 when we categorized the timing of the visit into four periods (never, <6, 6–12, and >12 months). When we dichotomized the timing of first predialysis nephrology care at >12 or ≤12 months, accuracy was 70% (κ=0.36), but it differed by patient characteristics and declined over time. In conclusion, we found substantial disagreement between information from the CMS Medical Evidence Report and Medicare physician claims on the timing of first predialysis nephrologist care. More-specific instructions may improve reporting and increase the utility of form CMS-2728 for research and public health surveillance.

Earlier care of patients with CKD approaching ESRD has been shown to be associated with improved outcomes during and after their transition to either transplantation or dialysis. Patients with CKD who saw a nephrologist earlier were more likely to initiate treatment using peritoneal dialysis and less likely to require a subsequent switch to in-center hemodialysis. ^{1 , 2} Earlier nephrology care has also been associated with increased access to kidney transplantation, a critically important fact, because waiting times are >5 years in most regions and are steadily increasing. ^{3 , 4} Earlier nephrology care has also been associated with lower mortality after initiation of dialysis. ^{5 – 10} In light of these potential benefits of earlier nephrologist care, clinical practice guidelines have recommended that patients with CKD be referred to nephrologists at the latest if their estimated GFR is <30 ml/min per 1.73 m². ¹¹

In 2005, the Centers for Medicare and Medicaid Services (CMS) revised the Medical Evidence Report (form CMS-2728), a mandatory form to be submitted to CMS by a patient’s dialysis provider to indicate incident ESRD among Medicare beneficiaries and to provide information to CMS for non-Medicare beneficiaries with ESRD who may then become eligible for Medicare coverage. Among the new items on form CMS-2728, information was requested on the presence and timing of the patient’s earliest nephrology care before start of dialysis.

The recently published 2020 Healthy People documents included a new item, CKD-10, with the objective to “increase the proportion of CKD patients receiving care from a nephrologist at least 12 months before the start of renal replacement therapy.” ¹² These data would be drawn from form CMS-2728 and the objective would be met if the proportion of patients with >12 months of predialysis nephrology were increased by 10% from 27.1% (2007) to 29.8%. Previous studies suggested that the accuracy of reporting of selected items on form CMS-2728 was relatively poor. ^{13 , 14} To test the accuracy of recorded information on the timing of nephrologist care, we conducted a validation study comparing providers’ responses on form CMS-2728 with submitted claims from physician visits in older patients who had Medicare coverage before onset of ESRD.

Results

Patient characteristics are summarized in Table 1. We identified 175,515 patients from the US Renal Data System (USRDS) who initiated dialysis between July 2005 and December 2008, and who were aged at least 67 years at the time of their first ESRD service. Among those patients, 80,509 (45.9%) had at least 24 months of uninterrupted Medicare (Part A and B) coverage as their primary payer before their first dialysis service date. Mean age was 76.7 years, 53.5% were male, and 79.2% were white (Table 1). We identified at least one outpatient predialysis nephrologist visit in 53,596 patients (66.6%), with a median time from their first nephrologist consultation to first ESRD service of 558 days (Figure 1A); more generally, 71,959 patients (88.9%) had at least one visit with a nephrologist in the outpatient or inpatient setting, a median 386 days before initiation of dialysis (Figure 1B).

Table 1: Patient characteristics

Figure 1: Days from first recorded nephrologist visit to initiation of dialysis. Time from earliest recorded nephrology consultation to first ESRD service date for the (A) first outpatient nephrologist consultation or the (B) first nephrologist consultation (inpatient or outpatient, any; up to 2 years before dialysis initiation).

Table 2 presents a cross-tabulation of patients by the timing of predialysis nephrologist care as reported in form CMS-2728 and as ascertained from billing claims. Among those who were reported to never have been seen by a nephrologist before ESRD, 16% (23% when including inpatient encounters) had actually been seen by a nephrologist >12 months before their first ESRD date, and additional 5% (8% including inpatient encounters) between 12 and >6 months and 15% (54% including inpatient encounters) within 6 months. Using only outpatient nephrology visits, agreement between the two sources of ascertainment was present in only 48.2% of patients and the κ statistic was 0.29 (95% confidence interval [95% CI], 0.28–0.29), indicating poor to fair agreement between the two data sources. By collapsing these categories into a dichotomous categorization at 12 months as will be used by the Healthy People 2020 initiative, ¹² our results showed that the κ statistic was 0.36 (95% CI, 0.35–0.37; results for variable dichotomization at 6 months and for none versus any are available in the Supplemental Tables 1 and 2). Analyses using the date of the earliest inpatient or outpatient nephrologist visit, rather than the earliest visit in any setting, showed even worse results: agreement was present in 35.4% (κ=0.15; 95% CI, 0.14–0.15) for the four-level variable and in 66.8% when dichotomized at 12 months (κ=0.33; 95% CI, 0.32–0.33; detailed data in Tables 2 and 3).

Table 2: Cross-tabulation of timing of first nephrologist consultation: Medical Evidence Report versus claims data

Table 3: Cross-tabulation of first nephrologist consultation, dichotomized at 12 months: Medical Evidence Report versus claims data

Considering the claims data as the criterion standard, Table 4 reports the overall sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of form CMS-2728 reporting based on dichotomization at 12 months (data for dichotomization at 6 months and for none versus any predialysis nephrology care available in Supplemental Tables 3 and 4). Overall, the sensitivity of the Medical Evidence Report to accurately identify patients who had seen a nephrologist at least 12 months before their first ESRD service date was 49%, whereas specificity to accurately identify patients who had never visited a nephrologist or who had received predialysis nephrologist care within 12 months was 85%. The PPV, or proportion of patients who had truly seen a nephrologist >12 months among all who were reported to have done so, was 72%, whereas the NPV was 69%. Table 4 shows these test statistics across patient characteristics. Accuracy varied significantly by all patient characteristics studied: accuracy was higher with older age (a function of increasing specificity in older patients), among women, and varied substantially by race and underlying kidney disease. Although measures of classification were generally more favorable using the first outpatient nephrologist visit as the criterion standard, the associations among classification measures (accuracy) and demographic characteristics were quite similar to analyses that included inpatient or outpatient nephrologist visits (Table 4). Regardless of its definition, accuracy of reporting declined significantly from 2005 to 2008 (P<0.001 in both analyses).

Table 4: Sensitivity, specificity, PPVs, and NPVs across patient characteristics

Table 5 describes the association between patient characteristics and the degree of discrepant reporting as expressed in the number of categories detailed in the methods. In an analysis simultaneously evaluating the relations among patient characteristics including age, sex, race, and primary cause of kidney disease and reporting discrepancies, all of these factors except for age were identified as independent correlates of accuracy (Table 5). Age showed an independent association with discrepant reporting as well when any inpatient nephrology consultations were taken into account; on average, form CMS-2728 and claims data differed by roughly one-third of a unit (Table 5). More specifically, older patients were increasingly more likely to be reported as having had relatively earlier nephrologist care than actually received (those aged >90 years had a 0.08 increase in discrepancy relative to those aged 67–69 years). Otherwise, analyses only considering outpatient nephrologist visits or those counting such encounters in any setting were qualitatively similar. Men had more disagreement than women. Asians and Native Americans had fewer reporting discrepancies than whites, whereas blacks had a larger level of disagreement than whites. There was also significant heterogeneity in the number of discrepant categories reported by primary cause of ESRD. Individuals with diabetes as the presumed cause of ESRD had more disagreement than those without diabetes. Disagreement increased monotonically with each more recent incident year.

Table 5: Correlates of the magnitude of discrepant reporting of first nephrology care between the Medical Evidence Report and claims data

Discussion

Our findings indicate substantial disagreement between information from the Medical Evidence Report (CMS-2728) and Medicare physician claims on the timing of earliest predialysis nephrologist care, thus diminishing the utility of the widely used form CMS-2728 for research and public health surveillance and planning. Furthermore, accuracy of reporting differed by important patient characteristics, most importantly demographic factors, which may introduce bias in research and public health surveillance. Although one might have speculated that the quality of reporting would improve over time as practitioners became more accustomed to detailing patients’ predialysis nephrology care (and more familiar with the forms themselves), our data indicate that the accuracy of reporting such care declined over the first 4 years of its use. These findings present opportunities to improve reporting of this important metric through a variety of measures.

Although we have demonstrated relatively poor accuracy of reporting, we can only speculate about the possible reasons for such. In practice, nephrologists who supervise patients’ initiation of dialysis may not necessarily know the patient well, especially if dialysis initiation occurs, perhaps urgently, during a hospitalization. Even if the reporting nephrologist has known the patient for some time, the earliest encounter with a nephrologist may have been a visit by a different provider—within or outside of a practice group—during an earlier hospitalization and may not be known to the reporting nephrologist. Finally, although form CMS-2728 should ideally be completed and signed by the responsible nephrologist, depending on a variety of factors, the form may be completed all or in part by other dialysis unit personnel, such as a social worker, nurse, or unit administrator, who may not be made aware of the full medical history of the patient.

In addition to these practice-related issues with capturing the correct information, there might be some ambiguity on the form itself about what exactly should be reported. The specific question (“Prior to ESRD therapy […] Was the patient under the care of a nephrologist?”) or the corresponding instructions on page 4 of the form do not specify whether the earliest encounter refers to an outpatient or inpatient encounter or whether inpatient encounter with the same or another nephrologist should also be considered. It also does not specify whether a single or isolated past encounter would be sufficient to warrant reporting or whether only the duration of continuous or ongoing nephrology care should be reported. Our analyses comparing the accuracy of reporting on form CMS-2728 indicate that providers seem to refer to the earliest outpatient encounter in their reporting, as indicated by more favorable agreement with care patterns as billed. However, even using the earliest outpatient encounter as the reference standard yielded only weak agreement (κ=0.36).

Although accuracy of reporting on the form CMS-2728 was generally poor, there were apparent differences in the quality of reporting by the time point chosen for dichotomization. Accuracy was highest for the first question of the reporting item, in which providers were queried on whether any (versus no) nephrology care was rendered to the patient before the date of their first ESRD treatment (κ=0.44, Supplemental Table 2). However, once the timing of the first episode of care was taken into account, as queried on the second part of the reporting item, accuracy declined (κ=0.28 for dichotomy at 6 months, Supplemental Table 1; and κ=0.36 for dichotomy at 12 months, Table 2). Accurate information on the timing of the earliest nephrology care is important, however, because the focus of the corresponding Healthy People 2020 goal is not just any occurrence of predialysis nephrology care, but its timing at >12 months before initial ESRD treatment. From a clinical perspective, although a patient may have first been seen 5 or 10 days before initiation of chronic dialysis, and thus be considered “referred” to nephrology, little time for education or intervention is left, rendering reporting of any nephrology care without specifying its duration of questionable value.

Our findings on the discrepant reporting between the Medical Evidence Report and claims data have several important implications. First, use of this form to affirm or refute success of the Healthy People 2020 goal would be problematic. From claims data in older adults initiating dialysis, it would seem that the set benchmark of 29.8% of patients initiating treatment for ESRD having been seen by a nephrologist for >12 months may already have been reached. ¹⁰ In our study, 43% of patients had seen a nephrologist as outpatients >12 months before initiation of treatment for ESRD. For future use of the form in health services research, it might be useful to specify what qualifies as a first nephrologist visit. It would seem reasonable to restrict eligible visits to those in the outpatient setting where nephrologists can focus more on education about treatment options for renal replacement therapy (i.e., evaluation for living- and deceased-donor transplantation), dialysis modality (peritoneal versus hemodialysis), other palliative therapy, shared decision making, and preparation for the chosen modality (e.g., creation of vascular access) than on acute issues, perhaps unrelated to CKD, that occupy attention during an inpatient setting. Second, and importantly, in addition to the lack of concordance with claims data, our findings suggest differential misclassification of predialysis nephrologist care by important demographic characteristics, particularly notable given large observed differences in survival on dialysis by these same characteristics. ¹⁵ Inaccuracies in the ascertainment of predialysis care patterns by nephrologists should be taken into account when conducting research or surveillance studies, especially when focusing on racial or geographic disparities.

Our validation study has certain limitations. We were only able to validate the Medical Evidence Form in older adults with continuous coverage through fee-for-service Medicare for an extended period of time before ESRD; it is possible that the accuracy and concordance differ in younger populations or in patients with alternative or no insurance coverage. Because our “look-back window” in medical claims was limited to 2 years, we may have missed first nephrology visits that occurred more remotely. It can be expected, however, that much earlier visits with a nephrologist would trigger subsequent follow-up visits. If one assumes that follow-up with a nephrologist would occur at least annually, the earliest visit recorded in our database would still appear in the >12 month category and our validation exercise would remain unbiased. Figure 1 seems to confirm this assumption, with monthly rates of first nephrology encounters being higher in the more distant past and leveling off 1.5 years before initiation of dialysis. We used billing claims for physician services as the reference criterion, which may be inaccurately or incompletely filed or recorded and have not been validated. Whereas a prospective study may allow for validation of our approach, such data are currently not available and may be subject to other sets of biases and generalizability issues. Finally, our study does not take into account any subsequent nephrology encounters after the earliest ones captured and used for analysis. The presence of a billed visit merely represents occurrence of a single encounter, but does not indicate any particular quality or intensity of care during or after it. Previous studies using claims or reporting sources similar to form CMS-2728 (in the Dialysis Morbidity and Mortality Study Wave 2) have already established the associations among frequency of nephrology care and outcomes after initiation of treatment for ESRD. ^{4 – 6}

The CMS (along with the dialysis care community) has recognized the complexities of caring for patients new to dialysis. The ESRD Prospective Payment System (“bundling”) enacted in January 2011 provides a 51% differential in the monthly capitated payment for the care of patients in the first 120 days of dialysis care. As one possible intervention, the CMS could require enhanced accuracy and consistency of reporting on form CMS-2728 in order for providers to maintain or augment this 120-day incentive.

In conclusion, we found a new item on a mandatory form proposed to be used for the assessment of quality of care and goal setting for a national initiative to be poorly representative of actual care billed for by older patients nearing ESRD. Our study underscores the need to take into account these limitations in future public health surveillance and research that uses this item. In light of the importance of this information and the lack of any observed improvements in reporting over time, it would be desirable to enhance the clarity of the queried item and provide better guidance to providers in subsequent iterations of the form. In addition, the CMS should consider establishing reporting quality standards and monitor provider compliance through audits. These standards could be incorporated in the CMS ESRD Quality Incentive Program, or serve as a modifier of the current first 120-day multiplier in the Prospective Payment System, to incentivize higher reporting standards.

Concise Methods

Study Population

We used the USRDS to define a cohort of patients with incident ESRD from July 2005 to December 2008. We identified patients who were aged at least 67 years at the time of their first ESRD service and who had at least 24 months of uninterrupted Medicare (Part A and B) coverage as their primary payer before their first dialysis service date. Information relating to the first predialysis nephrologist visit was ascertained from two sources: form CMS-2728 and claims data preceding each patient’s first treatment date by up to 24 months. On form CMS-2728, item 18 asks the following: “Prior to ESRD therapy […] Was the patient under the care of a nephrologist?” The response options are “yes,” “no,” “unknown,” and “If yes, answer: 6–12 months; >12 months.” We assumed that a nephrologist consultation took place <6 months, but at least 1 day prior to ESRD if the first response option “yes” was chosen, but neither “6–12 months” nor “>12 months” were selected.

From predialysis claims, we identified those claims indicating a visit by a nephrologist, and among those, the earliest recorded claim. We conducted two lines of analyses: one focused on the earliest recorded outpatient nephrologist visit, whereas the other used the earliest recorded nephrologist visit in any clinical setting (inpatient or outpatient). Consistent with the categories offered on form CMS-2728, patients were recorded to have had their earliest nephrologist visit in one of the following categories: never, <6 months, 6–12 months, or >12 months before ESRD. We also collapsed these data into two clinically meaningful categories: <12 months (or never) versus ≥12 months as used by the Healthy People 2020 initiative. We conducted supplementary analyses by dichotomizing this information into <6 months (or never) versus ≥6 months as well as into any nephrology encounter versus none. Using form CMS-2728, we ascertained each patient’s age, sex, race (Asian, black, Native American, other, white), and reported primary cause of kidney disease (cystic kidney disease, diabetic nephropathy, GN, hypertensive kidney disease, other).

Statistical Analyses

To assess accuracy of medical reporting, we examined a cross-tabulation of nephrologist care data from the two sources and calculated the proportion of claims that were accurately reported, using the claims data as the criterion standard. In addition, we calculated the κ statistic as a measure of agreement. The κ statistic takes into account the agreement due to chance, and is positive when the observed agreement is greater than that due to chance. A κ statistic of zero would be expected by chance alone. Generally, a κ statistic of 0.4 is considered to represent fair concordance; κ statistics of 0.6 and 0.8 are considered to represent moderate and substantial concordance, respectively. κ statistics <0.4 are considered to represent fair to poor concordance. After dichotomizing timing of predialysis nephrologist care as described above, we also calculated the sensitivity, specificity, PPVs, and NPVs of medical evidence reporting. We tested whether accuracy of reporting varied by patient characteristics using chi-squared tests.

We assessed whether the magnitude of discrepancy between claims and form CMS-2728 reporting was associated with specific patient characteristics using linear regression techniques. Discrepancy was defined as the difference between the category recorded in the claims data and that recorded in the Medical Evidence Report (0, never; 1, within 6 months; 2, between 6 and 12 months; and 3, >12 months ago), in which discrepant categories ranged from −3 to 3. A negative value reflects situations in which predialysis visits were reported earlier in the Medical Evidence Report than they had actually taken place. Conversely, positive discrepancies indicate situations in which patients received care earlier than was reflected in the Medical Evidence Report, and a value of zero indicates perfect agreement between the claim and reporting sources. To relate patient characteristics to level of discrepancy, we regressed level of discrepancy on all patient characteristics of interest jointly in order to obtain adjusted estimates of association.

All analyses were performed using R software (R version 2.12.1; GNU Project). This work was approved by the independent review board of Stanford University School of Medicine and was reviewed and approved by an officer at the National Institute of Diabetes and Digestive and Kidney Diseases.

Disclosures

None.

Data reported herein were supplied by the USRDS. Interpretation and reporting of these data are the responsibility of the authors and in no way should be seen as official policy or interpretation of the US government.