AKI affects a significant proportion of children in the pediatric intensive care unit (PICU) and is independently associated with worsened morbidity and increased mortality. Unfortunately, timely recognition of AKI is hampered by classic functional markers such as serum creatinine and urine output, both highly confounded and delayed to change after injury (1). Extensive research efforts have been expended to discover and validate biomarkers that detect AKI before functional marker change and predict AKI severity.
Novel AKI biomarkers have not demonstrated consistent predictive performance outside of the neonatal cardiopulmonary bypass (CPB) surgery population. AKI biomarkers demonstrate exceptional performance in these patients, largely because of demographic homogeneity, lack of comorbidities, and a known onset and duration of ischemic injury (2,3). However, in noncardiac PICU patients, performance of these same biomarkers is variable, with area under the curve receiver-operating characteristic (AUC-ROC) values ranging from 0.54 to 0.85 (Table 1) (4–7). Our previous data indicate that children suffering AKI>48 hours after admission are at highest risk for poor clinical outcome, including prolonged length of stay, need for RRT, or mortality (5,8). Thus, identification of patients at risk of AKI early in their intensive care unit (ICU) admission may help clinicians optimize the timing of resuscitative and supportive therapies. Identification of patients at intermediate risk of disease is equally important. For the purpose of risk stratification, we derived and validated the empirical clinical model of renal angina for prediction of subsequent severe AKI (9). The renal angina index (RAI) is a composite of patient AKI risk and early signs of injury, created to risk stratify patients for whom biomarkers testing would be most optimal (Figure 1). In short, fulfillment of renal angina informs the pretest probability of any AKI diagnostic test. Just as in the scenario of a potential heart attack, in which troponin levels are tested in patients with cardiac angina (thereby optimizing its use using clinical context), we propose that in scenarios of potential kidney attacks, AKI biomarkers will have greater utility and predictive value in patients who have renal angina fulfillment. Repeated evidence highlights the erosion of troponin performance when measured in patients without chest pain or at low clinical risk of myocardial infarction from coronary disease (10–16). In addition, independent of troponin, the absence of cardiac angina carries high negative predictive value for the diagnosis of a heart attack (17,18). Capricious, undirected use of any biomarker degrades its effectiveness.
In this study, we test the hypothesis that plasma AKI biomarker incorporation into the RAI significantly improves prediction of subsequent severe AKI in patients with septic shock compared with either the clinical model or AKI biomarkers.
Materials and Methods
Included patients were admitted to 17 different PICUs across the United States between 2006 and 2011 and were enrolled in a study investigating the genomic profile of children admitted with a diagnosis of severe sepsis and/or septic shock (19). The patients were enrolled under informed consent for the genomics work, allowing measurement of the biomarkers. In the original study, blood samples were obtained within 24 hours of initial presentation to the PICU with septic shock. Patients who presented with preexisting CKD or immediately after cardiac surgery were excluded. The institutional review boards approved secondary use/analysis of biologic samples and clinical data. All patients in the study were aged 28 days to 12 years. The patients analyzed in this study formerly comprised one of the validation cohorts in the initial renal angina study (CCHMC Sepsis 2, Validation 3) (20).
We defined the following data collection points. Day 0 was the first calendar day of PICU admission. Day 3 consisted of the time period between 72 and 96 hours after PICU admission. Baseline data included demographic information, admission diagnoses, Pediatric Risk of Mortality (PRISM-II) scores (21), and the first serum creatinine measurement (SCr) of the PICU admission. Day 0 data used for determination of renal angina included use of vasopressor and inotropic support and mechanical ventilatory support. We recorded changes in kidney function, based on estimated creatinine clearance (eCrCl) calculated by the Schwartz equation (22), for determination of the RAI. We reviewed the electronic health record for the lowest SCr up to 3 months before PICU admission to establish a reference eCrCl. If no SCr was available, we used a reference eCrCl of 120 ml/min per 1.73 m2 (23).
All patients were classified on day 0 as fulfilling criteria for renal angina [i.e., ANG(+) versus ANG(−)] using the RAI. Renal angina describes risk of AKI as a balance between risk and injury such that less early injury is required if a patient is at high risk for AKI compared with patients who are at low risk (Figure 1A). The RAI used to determine fulfillment or absence of renal angina is calculated as a composite of risk of AKI (based on published pediatric epidemiologic data) and initial injury based on changes from baseline in creatinine clearance and percent fluid overload (Figure 1B). A RAI score of ≥8 defined ANG(+) (20). Sensitivity analyses and breakdown of RAI derivation are presented in Supplemental Material A. Only eCrCl was used as an injury threshold for RAI calculation because fluid balance data were not available for these patients.
Plasma neutrophil gelatinase–associated lipocalin (NGAL), matrix metalloproteinase-8 (MMP-8), and neutrophil elastase-2 (Ela-2) were evaluated using xMAP multiplex array system technology (Luminex, Austin, TX). We chose these three biomarkers because NGAL is a well studied AKI biomarker, and MMP-8 and Ela-2 were identified as potential AKI biomarkers in the related genomic profile study (7,24). The biomarker measurements were performed before use in this study and investigators measuring these biomarker levels had no knowledge of the patient outcomes in the original study. The a priori determined cut-off values for each biomarker were determined by sensitivity analyses in the initial data sets. The cut-offs for MMP-8 and Ela-2 were ≥11 and ≥235 ng/ml, respectively (24). The cut-off used for serum NGAL was ≥150 ng/ml (7). Per the conditions of the original study, biomarkers were measured only in the first 24 hours of ICU admission. Blood was not drawn for testing after the first 24 hours.
The primary outcome was presence of severe AKI on day 3 of PICU admission (day 3 AKI). The presence of AKI on days 1 and 2 was not considered a positive AKI outcome. The presence of AKI on day 3, denoted as subsequent severe AKI, was considered a positive outcome. Day 3 was chosen because most PICU patients develop AKI within this timeframe, the time highlights the advantage of biomarkers on the PICU admission day to predict the outcome, and day 3 is a clinically relevant time frame for AKI management (20). Day 3 is also beyond the period of time that AKI may be functional and is more likely to be damage-associated AKI (25). Severe AKI was defined by the Kidney Disease Improving Global Outcomes AKI classification stage≥2: serum creatinine of 200% baseline (a decrease in eCrCl of ≥50% from baseline) (26). Urine output was not available in these patients. Secondary outcomes were PICU length of stay and in-hospital mortality. Data on RRT use were unavailable.
All statistical analyses were performed using STATA version 12 (StataCorp, College Station, TX), SAS version 9.3 (SAS Institute, Cary, NC), and R version 2.14.1 (R Development Core Team, Vienna, Austria). Continuous variables were reported as the median with interquartile range and were compared using the Mann–Whitney test. Categorical variables were summarized using frequency and proportion and compared by chi-squared or Fisher’s exact tests. An RAI cut-off of ≥8 was used to define renal angina fulfillment [ANG(+)] and this cut-off was used for operative characteristics (20). Simple and multivariable logistic models were used to predict day 3 AKI using RAI and its combinations with AKI biomarker(s) concentration levels. Area under the curve (AUC) values were calculated for each prediction model (RAI and biomarker concentrations used as continuous variables) and compared using DeLong’s method (27). Besides the increase in AUC, the performance improvement of the prediction models with AKI biomarkers added were also evaluated by the Akaike information criterion (AIC), net reclassification improvement (NRI), and integrated discrimination improvement (IDI) (28,29). No risk categories were selected for the calculation of NRI. R package Hmisc was used for the calculation of NRI and IDI. In all analyses, a P value <0.05 was considered statistically significant.
Demographics are shown in Table 2. The incidence of severe AKI at 3 days was 13.6% (29 of 214) in all patients, although the incidence was significantly higher in patients with RAI≥8 [ANG(+)] on admission (27 of 145 [18.6%] versus 2 of 69 [2.9%]; P=0.003). Age and PRISM-II scores were evaluated using multivariable logistic regression. Although younger age was independently associated with day 3 AKI, PRISM-II on admission was not. Fulfillment of renal angina was an independent predictor of day 3 AKI (Table 3).
Novel AKI Biomarkers Are Not Superior to the RAI for Prediction of Severe AKI
The comparison of AUC point estimates taken on day 0 showed that all of the plasma biomarkers demonstrated equivalent performance and none of them were superior to the RAI for prediction of day 3 AKI (Table 4).
Incorporation of AKI Biomarkers Augments the Predictive Performance of the RAI
The probability of AKI was calculated by determining the rate of AKI per subpopulation of biomarker and RAI combination, demonstrating a marked increase after incorporation of each biomarker (Table 5). Supplemental Material B depicts a patient tree, separating patients based on a priori biomarker decision rules. The predictive performance of RAI for day 3 AKI improved with the incorporation of each plasma biomarker separately (Figure 2, Table 6). Improvement in RAI predictive performance was greatest when both NGAL and Ela-2 were added (AUC=0.88; P value for increase from RAI AUC=0.02). The quality of the models (AIC) also improved with the incorporation of the biomarkers, decreasing by almost 13 points with the inclusion of NGAL and Ela-2 to the RAI (from 135.39 to 123.45).
AKI Biomarkers Augment the Classification Accuracy of AKI Prediction
Inclusion of all biomarkers, individually and in combinations of two, improved the classification accuracy of AKI prediction (Table 7). Although the NRI conferred by biomarker incorporation was most augmented by reclassification of nonevents (i.e., patients without day 3 AKI), improvement was also shown with event reclassification (i.e., patients with day 3 AKI). Weighted reclassification, measured by IDI demonstrated statistical significance with the incorporation of Ela-2 alone or with a combination of two biomarkers and the RAI.
In this study, we confirm the hypothesis that incorporation of biomarkers into a risk-stratified clinical model for AKI improves predictive performance for subsequent severe injury. The data presented are directly complementary to, and a logical follow-up of, our original analysis of renal angina. In the initial study, we demonstrated that the renal angina construct was a valid screening model to identify patients at highest risk for AKI. In this study, we demonstrate that a positive biomarker result in the population with an RAI≥8 significantly increases the predictive discrimination for subsequent severe AKI. Although the clinical management of AKI in critically ill children who are either premoribund or obviously recovering is unlikely to change based on the assessment of renal angina, we propose that this systematic approach to evaluating AKI risk carries the greatest clinical relevance for patients with an uncertain trajectory of illness. RAI assessment is early (day of admission), easy to practice (calculating RAI and measuring biomarkers is relatively simple compared with calculation of severity of illness scores), useful (high negative predictive value), and could potentially change initial resuscitation targets (fluid goals), use of medications (nephrotoxins), and timing of adjunctive therapies (RRT).
Judicious assessment of AKI biomarkers will increase their predictive performance and clinical effectiveness (30,31). AKI biomarkers need to demonstrate the appropriate balance of diagnostic performance and cost-effectiveness in order to gain widespread acceptance leading to implementation at the bedside. Indiscriminate biomarker measurement in every patient, regardless of size, age, and comorbidities, for a given injury or syndrome will render any test virtually useless. We have used the performance of troponin levels to detect acute coronary syndrome to provide an apt example of directed, optimized biomarker testing. Troponin measured in patients who exhibit cardiac angina, a combination of clinical signs, and known coronary disease risk factors allows practitioners to “rule in” heart attack. Troponin loses specificity when measured in patients without cardiac angina risk stratification (12). Unfortunately, unlike a heart attack, AKI does not carry an easily identifiable physical prodrome like cardiac angina. Simply put, a kidney attack does not “hurt” (32). Therefore, in order to optimize performance, care providers must seek novel ways of directing AKI biomarker use. The results of this proof-of-concept analysis demonstrate that incorporation of a positive AKI biomarker test (e.g., a renal troponin) in the novel RAI AKI risk stratification system (kidney pain) heightens the sensitivity and discrimination for a kidney attack.
Improvement of AKI prediction by incorporation of biomarkers into the RAI occurs via correct classification of disease. A recent study from our center demonstrated that each of these urinary biomarkers (NGAL, kidney injury molecule-1, IL-18, and liver-type fatty acid binding protein) added alone and in combination with the clinical model of patient age and CPB duration, improved prediction of AKI development and severity (33). The results presented here parallel the findings of this CPB study, demonstrating improvement in discrimination, evident by higher AUC (i.e., c-statistic) values, for AKI after incorporation of biomarkers. AUC is affected by sample and effect sizes. The metric of AIC evaluates the statistical weight of loss of information due to sample size. The lower AIC upon incorporation of AKI biomarkers into renal angina indicates that less information was lost when the goodness of fit of the AUC was balanced with the complexity of the model (Table 6). The NRI and IDI metrics recalibrate the AUC for weighted reclassification of both sample size and effect size, leading to a more accurate assessment of predictive and discriminative ability of a model (34). The significant changes in NRI and IDI after incorporation of biomarkers into the renal angina model occurred via weighting of the negative effect size such as the nonevents (i.e., no day 3 AKI) (Table 7). Finally, just as in the troponin paradigm, biomarker positivity in patients without renal angina does not improve prediction of day 3 AKI. The rate of AKI in patients with RAI<8 was 3%. A positive biomarker in those patients only increased the probability of AKI from 4.3% to 5% (Supplemental Material B).
Risk stratification using the RAI provides a clinically relevant and potentially useful tool to guide care. By quickly calculating the RAI and detecting renal angina, a clinician can identify the absence or fulfillment of renal angina in any patient on admission and then appropriately allocate the use of an AKI biomarker test to those in whom the test may yield the greatest predictive benefit. The excellent negative predictive value (>90%) of the RAI allows a clinician to reliably rule out AKI outside the window of functional AKI (25) and may afford more freedom of management for the acutely ill patient (i.e., higher volume resuscitation) with fewer concerns about developing the severe fluid overload consistently associated with poor outcomes in critically ill children (35–37).
Our study has several strengths. We examined a large multicenter cohort of critically ill children, overcoming potential single-center bias for initiation of mechanical ventilation and/or inotropic support. Although all participants were included in an original study of sepsis, our study includes a wide-ranging array of patient age and illness severity. Compared with previous work (Table 1), our study demonstrates a significant improvement in AKI prediction using novel biomarkers outside of the CPB population. Although RAI was not tested on ICU admission days other than day 0 or to predict AKI persistence before or after ICU day 3, the timeframe chosen is clinically relevant with regard to discriminating between reversible (functional) and damage-associated AKI (25).
There are several potential limitations to our study. The retrospective analysis of patients is subject to the inherent limitations of data extraction, correlative analyses, and conclusions associated with data not collected with the exclusive purpose of testing our hypothesis. We did not have access to fluid balance data or urine output data, both incorporated in the initial renal angina study and used as a metric of AKI outcome. In the initial validation of the RAI, the additive effect of fluid overload to the model was minimal (20). In addition, because the population studied in this analysis was a validation cohort used in the original study, we did not feel that it was prudent to recalibrate the RAI due to the lack of fluid data. We chose to study patients at day 3 because of the clinical significance of AKI lasting longer than the traditional reversible period. We did not extend the AKI analysis past 3 days because the number of patients remaining in the ICU until day 7 was too small to afford meaningful statistical results. The paucity of high-risk renal angina strata patients in our cohort (history of transplant) supports the call for larger pediatric population database studies. We did not have data regarding the use of RRT because the data studied were taken from a study directed at deriving biomarkers for sepsis and sepsis-associated morbidity (not for management variables). Of the biomarkers presented here, only NGAL is traditionally considered an AKI biomarker and was studied secondary to previous experience investigating NGAL as a septic shock–associated AKI biomarker in the pediatric population (7). Ela-2 and MMP-8 carry mild biologic plausibility for involvement in sepsis-associated AKI, but have not otherwise been described or analyzed as AKI biomarkers aside from our initial derivation study. We did not have urine AKI biomarkers available for this study, which could yield different performance after incorporation in the clinical model.
We conclude that risk stratification using the RAI in conjunction with AKI biomarker measurement augments prediction of subsequent severe AKI. We have now derived and validated the RAI in multiple populations from multiple medical centers and demonstrated initial proof-of-concept data—biomarker incorporation into a risk-stratified model bolsters prediction of AKI. The renal angina concept will require further, prospective analysis and may require recalibration in the lower-risk populations. Context provided by the RAI to direct testing of biomarkers may help differentiate damage-associated AKI from functional AKI (25). We believe that the RAI is a clinical adjunct that will lead to the optimization of AKI biomarker performance across the wide-ranging heterogeneity that exists across the general pediatric PICU population.
L.S.C. is a consultant for Alere, Abbott, and Astute.
We thank the following investigators that contributed biological samples and patient data for the database: Natalie Z. Cvijanovich (Children’s Hospital Oakland), Mark Hall (Nationwide Children’s Hospital), Geoffrey L. Allen (Children’s Mercy Hospital), Neal J. Thomas (Hershey Children’s Hospital), Robert J. Freishtat (Children’s National Medical Center), Nick Anas (Children’s Hospital of Orange County), Keith Meyer (Miami Children’s Hospital), Paul A. Checchia (Texas Children’s Hospital), Richard Lin (Children’s Hospital of Philadelphia), Michael T. Bigham (Akron Children’s Hospital), Anita Sen (Morgan Stanley Children’s Hospital), Jeffrey Nowak (Children’s Hospital and Clinics of Minnesota), Michael Quasney (Children’s Hospital of Wisconsin), Jared W. Henricksen (St. Christopher’s Hospital for Children), Arun Chopra (C.S. Mott Children’s Hospital), Sharon Banschbach (Cincinnati Children's Hospital Medical Center), Eileen Beckman (Cincinnati Children's Hospital Medical Center), Kelli Harmon (Cincinnati Children's Hospital Medical Center), Patrick Lahni (Cincinnati Children's Hospital Medical Center), and Thomas P. Shanley (C.S. Mott Children’s Hopsital).
Patients were enrolled in a study supported by grants from the National Institutes of Health (R01-GM064619, RC1-HL100474, and R01-GM099773).
Published online ahead of print. Publication date available at www.cjasn.org.
This article contains supplemental material online at http://cjasn.asnjournals.org/lookup/suppl/doi:10.2215/CJN.09720913/-/DCSupplemental.
See related editorial, “Renal Angina: Right Concept…Wrong Name?,” on pages 633–634.
1. McCullough PA, Shaw AD, Haase M, Bouchard J, Waikar SS, Siew ED, Murray PT, Mehta RL, Ronco C: Diagnosis of acute kidney injury using functional and injury biomarkers: Workgroup statements from the tenth Acute Dialysis Quality Initiative Consensus Conference. Contrib Nephrol 182: 13–29, 2013
2. Kwiatkowski DM, Goldstein SL, Krawczeski CD: Biomarkers of acute kidney injury in pediatric cardiac patients. Biomarkers Med 6: 273–282, 2012
3. Parikh CR, Devarajan P, Zappitelli M, Sint K, Thiessen-Philbrook H, Li S, Kim RW, Koyner JL, Coca SG, Edelstein CL, Shlipak MG, Garg AX, Krawczeski CDTRIBE-AKI Consortium: Postoperative biomarkers predict acute kidney injury and poor outcomes after pediatric cardiac surgery. J Am Soc Nephrol 22: 1737–1747, 2011
4. Herrero-Morín JD, Málaga S, Fernández N, Rey C, Diéguez MA, Solís G, Concha A, Medina A: Cystatin C and beta2-microglobulin: Markers of glomerular filtration in critically ill children. Crit Care 11: R59, 2007
5. Washburn KK, Zappitelli M, Arikan AA, Loftis L, Yalavarthy R, Parikh CR, Edelstein CL, Goldstein SL: Urinary interleukin-18 is an acute kidney injury biomarker in critically ill children. Nephrol Dial Transplant 23: 566–572, 2008
6. Zappitelli M, Washburn KK, Arikan AA, Loftis L, Ma Q, Devarajan P, Parikh CR, Goldstein SL: Urine neutrophil gelatinase-associated lipocalin is an early marker of acute kidney injury in critically ill children: A prospective cohort study. Crit Care 11: R84, 2007
7. Wheeler DS, Devarajan P, Ma Q, Harmon K, Monaco M, Cvijanovich N, Wong HR: Serum neutrophil gelatinase-associated lipocalin (NGAL) as a marker of acute kidney injury in critically ill children with septic shock. Crit Care Med 36: 1297–1303, 2008
8. Akcan-Arikan A, Zappitelli M, Loftis LL, Washburn KK, Jefferson LS, Goldstein SL: Modified RIFLE criteria in critically ill children with acute kidney injury. Kidney Int 71: 1028–1035, 2007
9. Goldstein SL, Chawla LS: Renal angina. Clin J Am Soc Nephrol 5: 943–949, 2010
10. Meyer M, Fink C, Roeger S, Apfaltrer P, Haghi D, Kaminski WE, Neumaier M, Schoenberg SO, Henzler T: Benefit of combining quantitative cardiac CT parameters with troponin I for predicting right ventricular dysfunction and adverse clinical events in patients with acute pulmonary embolism. Eur J Radiol 81: 3294–3299, 2012
11. Lim W, Whitlock R, Khera V, Devereaux PJ, Tkaczyk A, Heels-Ansdell D, Jacka M, Cook D: Etiology of troponin elevation in critically ill patients. J Crit Care 25: 322–328, 2010
12. Stein R, Gupta B, Agarwal S, Golub J, Bhutani D, Rosman A, Eng C: Prognostic implications of normal (<0.10 ng/ml) and borderline (0.10 to 1.49 ng/ml) troponin elevation levels in critically ill patients without acute coronary syndrome. Am J Cardiol 102: 509–512, 2008
13. Agewall S, Giannitsis E, Jernberg T, Katus H: Troponin elevation in coronary vs. non-coronary disease. Eur Heart J 32: 404–411, 2011
14. Agewall S, Tjora S: Physical exertion may cause high troponin levels. Tidsskr Nor Laegeforen 131: 2226, 2011
15. Tjora S, Gjestland H, Mordal S, Agewall S: Troponin rise in healthy subjects during exercise test. Int J Cardiol 151: 375–376, 2011
16. Ammann P, Fehr T, Minder EI, Günter C, Bertel O: Elevation of troponin I in sepsis and septic shock. Intensive Care Med 27: 965–969, 2001
17. Bassan R, Pimenta L, Scofano M, Gamarski R, Volschan AChest Pain Project investigators: Probability stratification and systematic diagnostic approach for chest pain patients in the emergency department. Crit Pathw Cardiol 3: 1–7, 2004
18. Bassan R, Pimenta L, Scofano M, Soares JFChest Pain Project Investigators: Accuracy of a neural diagnostic tree for the identification of acute coronary syndrome in patients with chest pain and no ST-segment elevation. Crit Pathw Cardiol 3: 72–78, 2004
19. Wong HR, Salisbury S, Xiao Q, Cvijanovich NZ, Hall M, Allen GL, Thomas NJ, Freishtat RJ, Anas N, Meyer K, Checchia PA, Lin R, Shanley TP, Bigham MT, Sen A, Nowak J, Quasney M, Henricksen JW, Chopra A, Banschbach S, Beckman E, Harmon K, Lahni P, Lindsell CJ: The pediatric sepsis biomarker risk model. Crit Care 16: R174, 2012
20. Basu RK, Zappitelli M, Brunner L, Wang Y, Wong HR, Chawla LS, Wheeler DS, Goldstein SL: Derivation and validation of the renal angina index to improve the prediction of acute kidney injury in critically ill children [published online ahead of print September 18, 2013]. Kidney Int doi:10.1038/ki.2013.349
21. Pollack MM, Ruttimann UE, Getson PR: Pediatric risk of mortality (PRISM) score. Crit Care Med 16: 1110–1116, 1988
22. Schwartz GJ, Work DF: Measurement and estimation of GFR in children and adolescents. Clin J Am Soc Nephrol 4: 1832–1843, 2009
23. Zappitelli M, Parikh CR, Akcan-Arikan A, Washburn KK, Moffett BS, Goldstein SL: Ascertainment and epidemiology of acute kidney injury varies with definition interpretation. Clin J Am Soc Nephrol 3: 948–954, 2008
24. Basu RK, Standage SW, Cvijanovich NZ, Allen GL, Thomas NJ, Freishtat RJ, Anas N, Meyer K, Checchia PA, Lin R, Shanley TP, Bigham MT, Wheeler DS, Devarajan P, Goldstein SL, Wong HR: Identification of candidate serum biomarkers for severe septic shock-associated kidney injury via microarray. Crit Care 15: R273, 2011
25. Endre ZH, Kellum JA, Di Somma S, Doi K, Goldstein SL, Koyner JL, Macedo E, Mehta RL, Murray PT: Differential diagnosis of AKI in clinical practice by functional and damage biomarkers: Workgroup statements from the tenth Acute Dialysis Quality Initiative Consensus Conference. Contrib Nephrol 182: 30–44, 2013
26. KDIGO Work Group: KDIGO Clinical Practice Guideline for Acute Kidney Injury. Kidney Int Suppl 2: 1–138, 2012
27. DeLong ER, DeLong DM, Clarke-Pearson DL: Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics 44: 837–845, 1988
28. Pencina MJ, D’Agostino RB Sr, Steyerberg EW: Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med 30: 11–21, 2011
29. Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS: Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Stat Med 27: 157–172–212; discussion 207–212, 2008
30. Al-Ismaili Z, Palijan A, Zappitelli M: Biomarkers of acute kidney injury in children: Discovery, evaluation, and clinical application. Pediatr Nephrol 26: 29–40, 2011
31. Glassford NJ, Eastwood GM, Young H, Peck L, Schneider AG, Bellomo R: Rationalizing the use of NGAL in the intensive care unit. Am J Respir Crit Care Med 184: 142–, author reply 143, 2011
32. Kellum JA, Bellomo R, Ronco C: Kidney attack. JAMA 307: 2265–2266, 2012
33. Krawczeski CD, Goldstein SL, Woo JG, Wang Y, Piyaphanee N, Ma Q, Bennett M, Devarajan P: Temporal relationship and predictive value of urinary acute kidney injury biomarkers after pediatric cardiopulmonary bypass. J Am Coll Cardiol 58: 2301–2309, 2011
34. Pencina MJ, D’Agostino RB, Vasan RS: Statistical methods for assessment of added usefulness of new biomarkers. Clin Chem Lab Med 48: 1703–1711, 2010
35. Arikan AA, Zappitelli M, Goldstein SL, Naipaul A, Jefferson LS, Loftis LL: Fluid overload is associated with impaired oxygenation and morbidity in critically ill children. Pediatr Crit Care Med 13: 253–258, 2012
36. Valentine SL, Sapru A, Higgerson RA, Spinella PC, Flori HR, Graham DA, Brett M, Convery M, Christie LM, Karamessinis L, Randolph AGPediatric Acute Lung Injury and Sepsis Investigator’s (PALISI) NetworkAcute Respiratory Distress Syndrome Clinical Research Network (ARDSNet): Fluid balance in critically ill children with acute lung injury. Crit Care Med 40: 2883–2889, 2012
37. Goldstein SL, Somers MJ, Baum MA, Symons JM, Brophy PD, Blowey D, Bunchman TE, Baker C, Mottes T, McAfee N, Barnett J, Morrison G, Rogers K, Fortenberry JD: Pediatric patients with multi-organ dysfunction syndrome receiving continuous renal replacement therapy. Kidney Int 67: 653–658, 2005