OBJECTIVE: To develop and validate a maternal comorbidity index to predict severe maternal morbidity, defined as the occurrence of acute maternal end-organ injury, or mortality.
METHODS: Data were derived from the Medicaid Analytic eXtract for the years 2000–2007. The primary outcome was defined as the occurrence of maternal end-organ injury or death during the delivery hospitalization through 30 days postpartum. The data set was randomly divided into a two-thirds development cohort and a one-third validation cohort. Using the development cohort, a logistic regression model predicting the primary outcome was created using a stepwise selection algorithm that included 24-candidate comorbid conditions and maternal age. Each of the conditions included in the final model was assigned a weight based on its beta coefficient, and these were used to calculate a maternal comorbidity index.
RESULTS: The cohort included 854,823 completed pregnancies, of which 9,901 (1.2%) were complicated by the primary study outcome. The derived score included 20 maternal conditions and maternal age. For each point increase in the score, the odds ratio for the primary outcome was 1.37 (95% confidence interval [CI] 1.35–1.39). The c-statistic for this model was 0.657 (95% CI 0.647–0.666). The derived score performed significantly better than available comorbidity indices in predicting maternal morbidity and mortality.
CONCLUSION: This new maternal comorbidity index provides a simple measure for summarizing the burden of maternal illness for use in the conduct of epidemiologic, health services, and comparative effectiveness research.
LEVEL OF EVIDENCE: II
A developed and validated comorbidity index that predicts maternal morbidity performs in a superior fashion compared with existing comorbidity scores.
Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women's Hospital and Harvard Medical School, the Department of Anesthesiology, Critical Care, and Pain Medicine, Massachusetts General Hospital, Harvard Medical School, and the Department of Epidemiology, Harvard School of Public Health, Boston, Massachusetts; the Department of Anesthesiology, University of Michigan Health System, Ann Arbor, Michigan; and the Division of Reproductive Health, Centers for Disease Control and Prevention, Atlanta, Georgia.
Corresponding author: Brian T. Bateman, MD, MSc, Division of Pharmacoepidemiology & Pharmacoeconomics, Department of Medicine, Brigham & Women's Hospital, Division of Obstetric Anesthesia, Department of Anesthesia, Critical Care, and Pain Medicine, Massachusetts General Hospital, 1620 Tremont Street, Suite 3030, Boston, MA 02120; e-mail: email@example.com.
The Medicaid Analytic eXtract (MAX) pregnancy cohort was supported by the Agency for Healthcare Research and Quality (AHRQ) (Grant R01HS018533). Research reported in this publication was supported by the Eunice Kennedy Shriver National Institute of Child Health and Human Development of the National Institutes of Health under Award Number K08HD075831 (B.T.B.). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the Centers for Disease Control and Prevention.
The authors thank Helen Mogun for assistance with data analysis.
Financial Disclosure Dr. Hernandez-Diaz has consulted for Novartis, GSK-Biologics, and AstraZenaca for unrelated projects. The other authors did not report any potential conflicts of interest.
In epidemiologic and health services research, patients' comorbidities must be identified and accounted for in analyses to avoid confounding bias. In certain circumstances, it is useful to have an index that summarizes the burden of comorbidity into a single numerical score.1,2 The most widely used indices for this purpose are the Charlson Comorbidity Index and the Elixhauser comorbidity classification system and their adaptations.3–9 These indices together have been cited more than 1,000 times annually in the medical literature in recent years.2
These indices have been applied in many studies in obstetrics for the purpose of describing and adjusting for comorbidity10–19 despite having been developed for nonobstetric populations. The Charlson Comorbidity Index was developed to predict 1-year mortality in medical patients.3 The Elixhauser comorbidity measure was developed to predict length of stay, hospital charges, and in-hospital death in explicitly nonobstetric admissions.8 Both of these scoring systems lack obstetric conditions that are important determinants of maternal morbidity and mortality. Furthermore, those conditions that do apply to obstetric patients are not weighted to reflect the unique contribution they make to the particular constellation of complications that present in an obstetric setting.
Recently there has been a call by leaders in the field of obstetrics to expand research into the determinants of severe maternal morbidity and mortality.20 The development of a comorbidity score applicable to obstetric patients would provide an important tool for summarizing comorbid illness and confounding control in such research. Such an index has not, to our knowledge, been described previously.
The objective of this study was, therefore, to develop and validate a maternal comorbidity index to predict severe maternal morbidity, defined as the occurrence of acute maternal end-organ injury, or mortality.
PATIENTS AND METHODS
The study cohort was derived from the Medicaid Analytic eXtract, a health care use data set that contains information on Medicaid enrollment and use claims, and included 2000–2007 data. Pregnancies were identified within this cohort as previously described.21 The Medicaid Analytic eXtract data set contains information regarding inpatient admissions, outpatient visits, and outpatient pharmacy dispensing claims. To allow adequate measurement of maternal comorbidities and outcomes, the cohort was restricted to women who delivered in-hospital and were eligible for Medicaid continuously from 180 days before the estimated last menstrual period through either 30 days postpartum or date of death during the 30-day postpartum period. To ensure complete ascertainment of relevant claims, we further restricted our analysis to women with 28 days or more of enrollment each month and without limited benefits, private insurance, or certain state-specific managed care programs that may underreport claims to the Medicaid Analytic eXtract.21 The analytic cohort included 854,823 completed pregnancies. The use of this deidentified database for research was deemed not human subjects research by the Partners institutional review board.
The primary outcome for the study was defined as maternal end-organ injury or death during the delivery admission through 30 days postpartum. End-organ injury was identified by the presence of a diagnostic codes from the International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) indicating acute heart failure, acute renal failure, acute liver disease, acute myocardial infarction, acute respiratory distress syndrome or respiratory failure, disseminated intravascular coagulation or coagulopathy, coma, delirium, puerperal cerebrovascular disorders, pulmonary edema, pulmonary embolism, sepsis, shock, status asthmaticus, or status epilepticus (see Appendix 1 for ICD-9-CM diagnostic codes, available online at http://links.lww.com/AOG/A435).22–25 Date of death was defined using the Medicaid eligibility file. The secondary outcome for the study was maternal intensive care unit admission during the delivery hospitalization through 30 days postpartum.
Based on a review of the relevant literature and clinical plausibility,24 we formulated a list of maternal comorbidities that potentially confer increased risk of maternal morbidity and mortality as candidate predictors for the comorbidity index. We restricted our analysis to conditions that would likely be identifiable during the antepartum period up to the time of admission for delivery and not complications that develop during the delivery admission. We then queried maternal inpatient and outpatient claims for ICD-9-CM diagnoses indicating the presence of these conditions from the 180-day prepregnancy period through the delivery hospitalization (see Appendix 2 for diagnostic codes, available online at http://links.lww.com/AOG/A436). We defined presence of each condition as having one or more corresponding codes during this period. We represented three pairs of conditions as hierarchies in the coding scheme. Specifically, if a patient had codes for severe preeclampsia or eclampsia, we coded her as not having mild or unspecified preeclampsia even if she had codes for it. Similarly, we allowed a patient to be defined as having gestational diabetes only if she did not have codes for pre-existing diabetes and as having gestational hypertension only if she did not have codes for pre-existing hypertension or preeclampsia or eclampsia.
We randomly allocated two thirds of the cohort to a development sample (n=569,882) and one third to a validation sample (n=284,941). Using the development sample, we constructed a multivariable logistic regression model using a fully stepwise selection algorithm that requires covariates to have a P value ≤.05 for both entry and retention in the model. The dependent variable was the presence of maternal end-organ injury or death during the delivery hospitalization through 30 days postpartum. The candidate independent variables included the 24 maternal comorbidities defined (see Appendix 2, http://links.lww.com/AOG/A436) as well as maternal age categorized as younger than 19, 19–34, 35–39, 40–44, and older than 44 years at the time of the last menstrual period.
The final model included 20 maternal conditions and maternal age. Using the beta coefficients from the final logistic regression model, we applied the weighting rule described by Schneeweiss and colleagues.1 In this rule, conditions with a beta coefficient (which corresponds to the natural logarithm of the odds ratio) 0.15 or less are assigned a weight of zero and for each 0.3 increase in the beta coefficient, the weight assigned to individual conditions is increased by 1 point.
For each patient in the cohort, the presence or absence of each of the 20 comorbidities included in the final model was defined and each of these conditions was weighted as described previously. Patients' comorbidity index was then obtained by summing the weights for all comorbidities present and adding it to the relevant weight for the patients' age. Importantly, patients were allowed to receive weights for mild or unspecified preeclampsia only if they did not also have severe preeclampsia or eclampsia and gestational hypertension only if they did not also have pre-existing hypertension or preeclampsia or eclampsia present.
The performance characteristics of the newly derived score were then assessed using the validation cohort. A logistic regression model was constructed including the primary outcome as the dependent variable and the maternal comorbidity index as a continuous independent variable. The discrimination of the model was evaluated by calculating a c-statistic (the area under the receiver operating curve). The calibration of the score was assessed by plotting the observed risk of the primary outcome by the maternal comorbidity index (divided into seven categories: 0, 1–2, 3–4, 5–6, 7–8, 9–10, and greater than 10).
To test the generalizability of the score to other measures of severe maternal morbidity, each of the performance characteristics described previously was tested for the outcome of maternal intensive care unit admission during the delivery admission through 30 days postdelivery using the validation cohort.
For each patient in the validation cohort, using claims from both the inpatient and outpatient records assessed from the prepregnancy period through the delivery hospitalization, we calculated the Romano adaptation of the Charlson Comorbidity Index,5 the van Walraven numerical modification of the Elxihauser score,8 and the Combined Comorbidity Score, which includes conditions from the Charlson and Elixhauser measures in a single score.2 The discrimination of each of these scores for the study end points was tested by constructing a logistic regression model for each score with the outcome (primary and secondary were separately assessed) included as the dependent variable and the numerical score for each patient included as a continuous independent variable; the c-statistic for each score was then calculated.
We next compared the predictive performance of the maternal comorbidity index with the Romano/Charlson index, van Walraven/Elxihauser score, and the Combined Comorbidity Score using a reclassification measure. From the logistic regression models described previously, we defined the predicted probability of the primary outcome for each patient from each scoring system. We then defined four categories of predicted risk, less than 2%, 2% or greater to less than 5%, 5% or greater to less than 10%, and 10% or greater corresponding to thresholds that members of the study team deemed as defining low-, moderate-, high-, and very-high-risk groups, respectively. We created a series of tables with the categories of predicted risk for the maternal comorbidity index plotted against predicted risk for each of the three other comorbidity indices evaluated noting the observed risk in each cell. We then calculated the net reclassification improvement for the maternal comorbidity index compared with each of the alternative scores.26 The net reclassification improvement is calculated as [Pr(up|O=1)−Pr(down|O=1)]+[Pr(down|O=0)−Pr(up|O=0)], where O=1 if the primary study outcome occurred (maternal death or end-organ injury from the delivery admission through 30 days postpartum) and O=0 if the primary outcome did not occur; “up” and “down” are defined by whether an individual was reclassified into a higher or lower predicted risk category by the maternal comorbidity index.2,26 Thus, for example, “Pr(up|O=1)” defines the probability of being reclassified to a higher risk category if the outcome is present and “Pr(down|O=1)” is the probability of being reclassified in a lower risk group if the outcome occurs. Stated differently, the net reclassification improvement reflects the sum of correct reclassification by the new maternal comorbidity index (ie, moving patients who had the primary outcome into higher risk strata and those who did not have the outcome into lower risk strata) minus the sum of the incorrect reclassifications.2 A positive net reclassification improvement therefore indicates better predictive ability for the new score than the score to which it is being compared. The statistical significance of the net reclassification improvement for each comparison was determined using the test of the null hypothesis, net reclassification improvement=0, derived by Pencina.26
We identified 854,823 completed pregnancies for analysis. The cohort was randomly divided into a development cohort that included 569,882 pregnancies and a validation cohort that included 284,941 pregnancies. Overall, 9,901 (1.16%) of pregnancies were complicated by the primary study outcome, maternal end-organ injury or death during the delivery admission through 30 days postpartum (including 6,606 [1.16%] in the development cohort and 3,295 [1.16%] in the validation cohort). The secondary study outcome, maternal intensive care unit admission during the delivery hospitalization through 30 days postpartum, occurred in 2,451 (0.29%) of pregnancies.
Table 1 shows the frequency of each type of maternal end-organ injury, which in a composite make up the primary study outcome. The most common types of maternal end-organ injury identified included sepsis (0.26%), acute heart failure (0.23%), disseminated intravascular coagulation or coagulopathy (0.17%), acute respiratory distress syndrome or respiratory failure (0.16%), acute liver disease (0.13%), and pulmonary edema (0.12%).
Table 2 shows the distribution of each of the potential predictors stratified by whether the pregnancy was complicated by the primary study end point. All of the antepartum maternal conditions assessed were more common in women whose pregnancies were complicated by end-organ injury or death with the exception of tobacco use. Women who experienced the primary study outcome were slightly older than those who were unaffected.
The final model from the stepwise selection algorithm used to identify predictors of the primary study outcome is shown in Table 3 along with the assigned weights that make up the maternal comorbidity index. When the maternal comorbidity index was calculated for each patient in the validation cohort and a logistic regression model predicting the primary outcome was fitted including the score as a continuous independent variable, the odds ratio per point increase in the comorbidity index was 1.37 (95% confidence interval [CI] 1.35–1.39). The c-statistic for this model was 0.657 (95% CI 0.647–0.666) indicating moderate discrimination.
The observed risk of the primary outcome by category of maternal comorbidity index is shown in Figure 1. The risk increased from 0.68% in patients with a score of 0 to 10.9% in those with a score of greater than 10. The maximum range for the maternal comorbidity index is 0–45. The range for women in the validation cohort was 0–19. The mean score was 0.91 (standard deviation 1.42) and the median score was 0 (interquartile range 0–1).
The logistic regression model predicting the secondary outcome, maternal intensive care unit admission, yielded an odds ratio per point increase in the comorbidity index of 1.36 (95% CI 1.33–1.40). The c-statistic for this model was 0.651 (95% CI 0.631–0.670), again indicating moderate discrimination. The observed risk of the secondary outcome by category of index is shown in Figure 2. The risk increased from 0.18% in those with a score of 0 to 2.7% in those with a score of greater than 8 (observed risk at higher levels of score cutoff cannot be displayed owing to small cell sizes).
Table 4 shows the net reclassification statistics comparing the maternal comorbidity index to the other commonly used indices using the validation cohort. The net reclassification improvement for the maternal comorbidity index was 0.118 (P<.001) when compared with the Charlson/Romano Index, 0.071 (P<.001) when compared with the van Walraven/Elxihauser Score and 0.027 (P=.011) when compared with the Combined Comorbidity Score. We also calculated the c-statistic for each of the scores. For the primary outcome, the c-statistic for the Charlson/Romano Index was 0.578 (95% CI 0.570–0.585), for the van Walraven/Elxihauser Score was 0.586 (95% CI 0.575–0.597), and for the Combined Comorbidity Score was 0.617 (95% CI 0.608–0.627). For the secondary outcome, the c-statistic for the Charlson/Romano Index was 0.560 (95% CI 0.545–0.574), for the van Walraven/Elxihauser Score was 0.565 (95% CI 0.544–0.585), and for the Combined Comorbidity Score was 0.563 (95% CI 0.545–0.582). These findings suggest significantly improved discriminative ability of the maternal comorbidity index for the primary and secondary study end points.
We also tested the performance of more parsimonious implementations of the maternal comorbidity index. When we only included conditions with a weight of greater than one, the c-statistic for predicting the primary outcome in the validation cohort fell to 0.634 (95% CI 0.625–0.643), which compares with 0.657 (95% CI 0.647–0.666) for the full model. When we included only conditions with a weight greater than or equal to two, the c-statistic further fell to 0.607 (95% CI 0.599–0.615).
Using this cohort of 854,823 completed pregnancies for which inpatient and outpatient claims were available from 6 months before pregnancy through 30 days postpartum, we have developed and validated a simple numerical score that summarizes obstetric and medical comorbidities and predicts severe maternal morbidity and mortality. As epidemiologic, comparative effectiveness, and health services research aimed at improving maternal outcomes increasingly becomes a priority area in obstetric research,20 this comorbidity index promises to be an important tool for use in such work. Code to implement the index is freely available online at: http://www.drugepi.org/dope-downloads/.
Existing comorbidity scores such as the Charlson/Romano Index, the van Walraven/Elxihauser Score, or the Combined Comorbidity Score lack relevant obstetric conditions and use weighting schemes that are not necessarily applicable to obstetric outcomes. Nevertheless, a large number of studies in obstetrics have applied these nonobstetric scores when describing or adjusting or both for comorbidities. Our maternal comorbidity index performed significantly better in predicting the primary study outcome, acute maternal end-organ injury or death, than did these other scores as measured by the net reclassification improvement and c-statistic (measures of calibration and discrimination). For example, as compared with the commonly used Charlson Index, our maternal comorbidity index correctly reclassified 20.8% of women while incorrectly reclassifying only 9.0%, indicating improvement in classification for 11.8% of the validation cohort. More accurately measuring a strong risk factor for such a large portion of a study population can substantially reduce confounding bias in observational studies.27
Having such a summary score for use in obstetric research is particularly important given the relative infrequency with which severe maternal morbidity and mortality occurs in developed countries.23,24,28 Although ideally studies examining the effect of a risk factor or intervention on the risk of adverse maternal outcomes would measure and individually adjust for all relevant confounders using a regression model or the like, with infrequent outcomes, this is not always possible without overspecifying the model. In these circumstances, an aggregate measure of the burden of relevant comorbidities, weighted in a manner that is relevant to the outcome of interest, becomes a highly useful tool for confounding control.
The scoring system described here may have clinical use as well. It may be helpful in identifying patients who would benefit from consultation by a maternal-fetal medicine specialist before pregnancy, during pregnancy, or both. It may also be useful as a means of triaging patients to high-risk centers that have the infrastructure, staffing, and subspecialty expertise required to care for those whose score suggests significant risk for a complicated delivery or postpartum course. Leaders in obstetrics have recently called for the development of networks with hospitals designated by levels of maternal care.29 This tool may find application as a screening tool for ensuring that parturients are being cared for in an appropriate setting within such a network. Trials to test these and other possible applications of a maternal comorbidity index would provide important information to improve clinical practice.
The development and validation of this maternal comorbidity index is subject to certain limitations. The weights assigned to a particular comorbidity reflect the average population effect of that comorbidity on outcome. However, there is clearly a spectrum of severity for each of the conditions included in our analysis. Thus, although an inherent part of any such index, this may result in residual confounding when the tool is used for adjustment in research. It also means that practitioners need to exercise judgment if the score is applied in a clinical context (eg, a patient with pulmonary hypertension may have a score of 4, but, if the patient's pulmonary pressures are suprasystemic, the risk of adverse outcome may substantially exceed the average patient with a score of 4). An additional limitation is that the scoring system only achieves moderate discrimination for both the primary and secondary end point (c-statistic of 0.66 and 0.65, respectively). This likely reflects the fact that many etiologies of severe maternal morbidity, including postpartum hemorrhage and stroke,24,30,31 frequently occur in the absence of recognized risk factors. It may also reflect the fact that certain comorbidities, including maternal obesity, are not well coded in health care use data. To accurately identify pregnancies within the Medicaid Analytic eXtract, we require a linkage to neonatal records. This means that our cohort is limited to completed pregnancies; some women with severe systemic disease may elect to terminate pregnancy and would not be captured in our analysis. Additionally, maternal deaths are identified in the Medicaid Analytic eXtract using the eligibility file, which may underestimate deaths. However, because deaths represent only a small fraction of the composite end point and generally occur in the setting of measured maternal end-organ injury, this limitation is not likely to substantially alter our results. It is important to note that although we restricted the development and validation cohorts to those women with Medicaid coverage from 6 months antepartum through 30 days postpartum to maximize our measurement of comorbid illness, this coverage criterion is not necessary when applying the score in practice. Finally, our index was developed in a Medicaid population. Although we expect that the associations of maternal comorbidities to adverse outcomes defined in this study will hold in other groups, future studies will need to determine the extent to which the index needs to be tailored for optimal performance in other settings including the commercially insured population in the United States and pregnant populations in other countries. However, it is notable that the existing comorbidity scores have found wide application beyond the populations in which they were originally developed. Furthermore, because Medicaid covers nearly half of all deliveries in the United States, a scoring system optimized for this population is of substantial interest, particularly in light of the relatively high risk of maternal morbidity in this group.32
With the substantial rise in the incidence of severe maternal morbidity that has occurred over the past decade,23 there is an urgent need for research into the determinants of such morbidity and interventions aimed at preventing it. Much of this research has been, and will likely continue to be, based on administrative data. The maternal comorbidity index provides a simple approach to summarizing medical and obstetric comorbidities and can be used for confounding control in such research.
1. Schneeweiss S, Wang PS, Avorn J, Glynn RJ. Improved comorbidity adjustment for predicting mortality in Medicare populations. Health Serv Res 2003;38:1103–20.
2. Gagne JJ, Glynn RJ, Avorn J, Levin R, Schneeweiss S. A combined comorbidity score predicted mortality in elderly patients better than existing scores. J Clin Epidemiol 2011;64:749–59.
3. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 1987;40:373–83.
4. Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol 1992;45:613–9.
5. Romano PS, Roos LL, Jollis JG. Adapting a clinical comorbidity index for use with ICD-9-CM administrative data: differing perspectives. J Clin Epidemiol 1993;46:1075–9.
6. D'Hoore W, Sicotte C, Tilquin C. Risk adjustment in outcome assessment: the Charlson comorbidity index. Methods Inf Med 1993;32:382–7.
7. Ghali WA, Hall RE, Rosen AK, Ash AS, Moskowitz MA. Searching for an improved clinical comorbidity index for use with ICD-9-CM administrative data. J Clin Epidemiol 1996;49:273–8.
8. Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care 1998;36:8–27.
9. van Walraven C, Austin PC, Jennings A, Quan H, Forster AJ. A modification of the Elixhauser comorbidity measures into a point system for hospital death using administrative data. Med Care 2009;47:626–33.
10. Krieger EV, Landzberg MJ, Economy KE, Webb GD, Opotowsky AR. Comparison of risk of hypertensive complications of pregnancy among women with versus without coarctation of the aorta. Am J Cardiol 2011;107:1529–34.
11. Opotowsky AR, Siddiqi OK, D'Souza B, Webb GD, Fernandes SM, Landzberg MJ. Maternal cardiovascular events during childbirth among women with congenital heart disease. Heart 2012;98:145–51.
12. Nguyen GC, Boudreau H, Harris ML, Maxwell CV. Outcomes of obstetric hospitalizations among women with inflammatory bowel disease in the United States. Clin Gastroenterol Hepatol 2009;7:329–34.
13. Murthy SK, Heathcote EJ, Nguyen GC. Impact of cirrhosis and liver transplant on maternal health during labor and delivery. Clin Gastroenterol Hepatol 2009;7:1367–72,1372.e1.
14. Wright JD, Devine P, Shah M, Gaddipati S, Lewin SN, Simpson LL, et al.. Morbidity and mortality of peripartum hysterectomy. Obstet Gynecol 2010;115:1187–93.
15. Panchal S, Arria AM, Harris AP. Intensive care utilization during hospital admission for delivery: prevalence, risk factors, and outcomes in a statewide population. Anesthesiology 2000;92:1537–44.
16. Cheesman K, Brady JE, Flood P, Li G. Epidemiology of anesthesia-related complications in labor and delivery, New York State, 2002–2005. Anesth Analg 2009;109:1174–81.
17. Jin Y, Carriere KC, Marrie TJ, Predy G, Johnson DH. The effects of community-acquired pneumonia during pregnancy ending with a live birth. Am J Obstet Gynecol 2003;188:800–6.
18. Dolan GP, Myles PR, Brett SJ, Enstone JE, Read RC, Openshaw PJ, et al.. The comparative clinical course of pregnant and non-pregnant women hospitalised with influenza A(H1N1)pdm09 infection. PLoS One 2012;7:e41638.
19. Needleman J, Minnick AF. Anesthesia provider model, hospital resources, and maternal outcomes. Health Serv Res 2009;44:464–82.
20. D'Alton ME, Bonanno CA, Berkowitz RL, Brown HL, Copel JA, Cunningham FG, et al.. Putting the ‘M’ back in maternal-fetal medicine. Am J Obstet Gynecol 2013;208:442–8–.
21. Palmsten K, Huybrechts KF, Mogun H, Kowal MK, Williams PL, Michels KB, et al.. Harnessing the Medicaid Analytic eXtract (MAX) to Evaluate Medications in pregnancy: design considerations. PLoS One 2013;8:e67405.
22. Callaghan WM, Mackay AP, Berg CJ. Identification of severe maternal morbidity during delivery hospitalizations, United States, 1991–2003. Am J Obstet Gynecol 2008;199:133.e1–8.
23. Callaghan WM, Creanga AA, Kuklina EV. Severe maternal morbidity among delivery and postpartum hospitalizations in the United States. Obstet Gynecol 2012;120:1029–36.
24. Mhyre JM, Bateman BT, Leffert LR. Influence of patient comorbidities on the risk of near-miss maternal morbidity or mortality. Anesthesiology 2011;115:963–72.
25. Berg CJ, Mackay AP, Qin C, Callaghan WM. Overview of maternal morbidity during hospitalization for labor and delivery in the United States: 1993–1997 and 2001–2005. Obstet Gynecol 2009;113:1075–81.
26. Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008;27:157–72.
27. Schneeweiss S, Seeger JD, Maclure M, Wang PS, Avorn J, Glynn RJ. Performance of comorbidity scores to control for confounding in epidemiologic studies using claims data. Am J Epidemiol 2001;154:854–64.
28. Berg CJ, Callaghan WM, Syverson C, Henderson Z. Pregnancy-related mortality in the United States, 1998 to 2005. Obstet Gynecol 2010;116:1302–9.
29. Hankins GD, Clark SL, Pacheco LD, O'Keeffe D, D'Alton M, Saade GR. Maternal mortality, near misses, and severe morbidity: lowering rates through designated levels of maternity care. Obstet Gynecol 2012;120:929–34.
30. Bateman BT, Schumacher HC, Bushnell CD, Pile-Spellman J, Simpson LL, Sacco RL, et al.. Intracerebral hemorrhage in pregnancy: frequency, risk factors, and outcome. Neurology 2006;67:424–9.
31. Bateman BT, Berman MF, Riley LE, Leffert LR. The epidemiology of postpartum hemorrhage in a large, nationwide sample of deliveries. Anesth Analg 2010;110:1368–73.
32. Markus AR, Rosenbaum S. The role of medicaid in promoting access to high-quality, high-value maternity care. Women's Health Issues 2010;20:S67–78.