Preterm birth is defined as delivery before 37 weeks of gestation. Although it was a goal of Healthy People 20101 to decrease the preterm birth risk to approximately 7%, the national incidence of preterm birth remains greater than 12%.2 This is attributable to both our inability to accurately predict who will deliver preterm and our lack of universal efficacious regimens to treat preterm labor and prevent preterm birth.
Prematurity contributes to approximately 80% of perinatal mortality in the United States.3 Therefore, despite our inability to significantly decrease the rate of preterm birth, the ability to identify patients at greatest risk for imminent preterm birth nevertheless is valuable to provide medical management that may decrease the adversity associated with prematurity. Specifically, in addition to transfer to a tertiary care center, in patient observation, cerebral palsy prophylaxis,4 and tocolysis,5 the American College of Obstetricians and Gynecologists recommends the administration of corticosteroids to all women between 24 and 33 6/7 weeks of gestation at risk for delivering within 7 days in an effort to “reduce the risks of respiratory distress syndrome, perinatal mortality, and other morbidities.”6–9
Preterm labor, defined as cervical change in the setting of uterine contractions, is a risk factor for imminent preterm birth. However, only 20–30% of women with preterm labor deliver preterm.10 Although fetal fibronectin11 and transvaginal cervical length measurements12 are oftentimes used in an effort to risk-stratify patients symptomatic for preterm labor, such tools are expensive and require training and technology to use. Furthermore, they only can be used under certain clinical circumstances. Specifically, fetal fibronectin cannot be performed when the cervix is dilated 3 cm or more, when membranes have ruptured, or after a digital cervical examination has been performed.13–15 Cervical length measurement is not predictive of preterm birth risk when the cervix is dilated 2 cm or more, is inaccurate if not performed very precisely, and is generally most useful for predicting future rather than imminent preterm birth.12
A validated clinical predication rule that identifies which women symptomatic for preterm labor are at greatest risk for relatively imminent preterm birth through use of demographic and clinical risk factors alone could be of high clinical utility to allow for the most judicious use of resources while being straightforward and inexpensive to perform. Based on a PubMed search in February 2012 of articles published in the English language using MeSH “predict,” “preterm birth,” and “preterm labor,” to our knowledge, such a prediction rule in patients symptomatic for preterm labor has not been published.
Therefore, the primary objective of this study was to develop a prediction rule for delivery within 10 days of preterm labor using only readily obtainable demographic and clinical risk factors–including gestational age at presentation with preterm labor, obstetric history, prenatal care status, initial cervical dilatation, maternal age, obesity, race, and tobacco use–but not fetal fibronectin, transvaginal cervical length assessment, or serum biomarkers.
Although our primary aim was delivery within 10 days, there is also clinical value to identifying which women, presenting with preterm labor, will deliver before 37 weeks of gestation in an effort to ensure that these patients are more closely monitored and potentially offered corticosteroids to mitigate the risk of prematurity. Therefore, the secondary objective of this study was to develop a prediction rule for preterm birth before 37 weeks of gestation in a cohort of women symptomatic for preterm labor.
MATERIALS AND METHODS
We performed analyses of data that were collected for a prospective cohort study at a single urban tertiary care center. The cohort consisted of a sample of women with singleton pregnancies between 22 and 33 6/7 weeks of gestation who presented to the labor and delivery triage unit with symptoms concerning for preterm labor, including contractions, cramping, asymptomatic vaginal bleeding, vaginal pressure, and abdominal or back pain.
To have preterm labor diagnosed at our institution, a patient must make documented cervical change on sterile cervical examination in the presence of regular uterine contractions. Patients with preterm labor diagnosed between 24 0/7 and 33 6/7 weeks of gestation are admitted to the hospital and receive 48 hours of magnesium sulfate tocolysis for the administration of a single course (two doses) of betamethasone. Patients with ruptured membranes at less than 34 weeks of gestation before the onset of labor are managed conservatively with administration of a single course of betamethasone and 7 days of latency antibiotics. Patients with preterm premature rupture of membranes are delivered in the setting of chorioamnionitis, spontaneous labor, or achieving 34 weeks of gestation or more. Patients who present with symptoms concerning for preterm labor or preterm premature rupture of membranes who do not meet diagnostic criteria are discharged home with instructions to continue routine prenatal care and to return for further evaluation as needed. Enrollment to the parent study did not influence patient care in any way. Of note, we do not use fetal fibronectin or cervical length measurement in the diagnosis or management of the preterm labor patient.
A patient did not have to be admitted to the hospital with a diagnosis of preterm labor or preterm premature rupture of membranes to be a candidate for enrollment. For the current study, we included all eligible women from the parent study. Patients were excluded from the parent study and thus from these analyses for multiple gestation, major fetal anomaly, intrauterine fetal demise, severe preeclampsia before enrollment, chronic steroid or immunosuppressive drug use, active immunologic disease, acute systemic febrile illness, or pregestational diabetes. Patients with unknown delivery information were also excluded from these analyses.
Patients were enrolled in the study by trained clinical research coordinators who obtained informed consent at the time of enrollment. The clinical research coordinators enrolled consecutive patients during daytime hours Sunday through Friday and during evening hours Monday through Thursday. Once a patient was enrolled in the study, all management decisions were made by the treating physician according to the standard of care at our institution. Women were enrolled from April 2008 through December 2010.
After enrollment, each patient was tracked for the remainder of her pregnancy and relevant delivery information was obtained through chart review. Previously published studies have suggested that demographic risk factors such as maternal age,16,17 low body mass index,16,17 and obstetric history,18 as well as modifiable traits such as no prenatal care16,17 and tobacco use during pregnancy,19 might predispose to preterm birth. Therefore, this information and pertinent medical, surgical, and gynecologic histories were recorded. Initial cervical dilatation was also obtained (0 to less than 2 cm, 2 to less than 3 cm, 3 to less than 4 cm, 4 cm or more).
The primary outcome of these analyses was delivery within 10 days of presentation to the hospital with symptoms concerning for preterm labor. The secondary outcome of interest was preterm birth before 37 weeks of gestation.
Pearson χ2 analyses were used to determine associations between categorical risk factors and both the primary and secondary outcomes. Univariable logistic regression was performed to compute odds ratios along with 95% confidence intervals (CIs) to estimate the associations between the primary and secondary outcomes and each demographic and clinical risk factor.20
Variables identified as potential risk factors in unadjusted analyses (P<.2) were used to create separate multivariable logistic models for the primary and secondary outcomes.21 After starting with the most comprehensive model that included all potential risk factors, a backwards selection method was performed to determine which combination of risk factors generated the most parsimonious yet predictive model for each outcome.22 With the successive elimination of each variable from the model, the area under the curve (AUC) was compared with the AUC of the previous model containing the variable using a statistical receiver-operator curve area comparison test. If the P value describing the comparison of consecutive AUCs was not significant, then we concluded that removing the variable did not significantly reduce the predictive capability of the model.22 The AUC of each final multivariable model was then compared with the AUC generated by the univariable model that generated the strongest AUC using the same statistical receiver-operator curve area comparison test to ensure that the P value describing the comparison of these two AUCs was significant. In that case, we concluded that the multivariable model significantly improved the ability to predict each adverse obstetric outcome.
The odds ratios of each covariate in the final multivariable model were rounded to the nearest whole number. These rounded values were the estimated weights for each covariate that could be summed to generate a final score that might predict the probability of each outcome. A prediction score was calculated for each patient in the data set and used to determine the sensitivity, specificity, positive predictive value, and negative predictive value for a range of score cut points.23
For both delivery within 10 days and preterm birth at less than 37 weeks of gestation, the AUC of the score was compared with the AUC of the final multivariable logistic model using a statistical receiver-operator curve area comparison test to ensure that the P value describing the comparison of these AUCs was not significant. In that case, we concluded that there was no statistically significant difference between the predictive ability of the score and the final multivariable model. The score that generated the highest negative predictive value was defined as a positive test result for each outcome.
Finally, bootstrapping techniques with 1,000 replications were performed to internally validate the scoring system to estimate 95% CIs for the performance characteristics.24 STATA 10.1 was used for data analysis. In all analyses, P<.05 was considered statistically significant.
A priori sample size calculations were computed to allow for a specified CI around the sensitivity of prediction.25 For the calculation, we assumed our prediction rule would have at least 65% sensitivity for the primary outcome. A 95% CI with a precision of 10% would require for us to observe 88 deliveries occurring within 10 days of presentation. Based on nonpublished data from our institution, we assumed that the prevalence of delivery within 10 days of presentation with preterm labor was approximately 15%. Given that the parent study recruited only women who planned to deliver at our institution, we assumed that a 5% loss to follow-up would be reasonable. Therefore, assuming a 15% prevalence of delivery within 10 days of presentation, a two-sided type I error of 0.05, and allowing for 5% of patients to be lost to follow-up, we estimated that we would need to enroll approximately 614 patients to obtain data from 583 women for analysis. This study was approved by the Institutional Review Board at the University of Pennsylvania.
The actual loss to follow-up during the study period was slightly higher than we anticipated (8.0%, not 5.0%). Therefore, we enrolled a total of 634 women into the cohort to achieve our desired final cohort size of 583 women. Consistent with a high-risk cohort, the prevalence of delivery within 10 days of initial presentation with symptoms of preterm labor was 15.4% (95% CI 12.6–18.6; n=90). The prevalence of preterm birth at less than 37 weeks of gestation was 35.0% (95% CI 31.1–39.0; n=204).
The associations between demographic variables and both the primary and secondary outcomes were studied to identify potential predictors. No prenatal care, initial cervical dilatation, and tobacco use during pregnancy were identified as potential predictors of delivery within 10 days (Table 1). No prenatal care, initial cervical dilatation, tobacco use, obstetric history, and African American race were identified as potential predictors of delivery at less than 37 weeks of gestation (Table 1). There was no significant difference between gestational age at presentation with preterm labor and either outcome (delivery within 10 days: P=.98; preterm birth at less than 37 weeks of gestation: P=.24).
The associations between the primary or secondary outcomes and each individual demographic risk factor are summarized in Table 2. The ability of each individual risk factor to predict the primary or secondary outcomes was quite poor, generating AUCs between 0.47 and 0.57. Only initial cervical dilatation was associated with a somewhat strong discriminatory ability to predict each outcome (delivery within 10 days: AUC=0.71; preterm birth at less than 37 weeks of gestation: AUC=0.70). The test characteristics of initial cervical dilatation to predict each outcome are depicted in Table 3.
The initial model to predict delivery within 10 days included all potential risk factors identified through unadjusted analyses (initial cervical dilatation, no prenatal care, and tobacco use). Performing comparative AUC analyses after serial backwards elimination confirmed that removing any risk factor significantly reduced the predictive capability of the model. Therefore, the initial and final models for the primary outcome were the same. This multivariable model was significantly more predictive of delivery within 10 days than initial cervical dilatation alone, with the univariable model having the greatest individual predictive ability (AUC=0.75 compared with 0.71, P=.009; Fig. 1).
The adjusted odds ratios of each risk factor in the final multivariable model for delivery within 10 days along with the weighted estimate assigned to each risk factor are found in Table 4. The total score to predict delivery within 10 days can be expressed as: 4×(no prenatal care)+0×(initial cervical dilatation: 0 to less than 2 cm)+2×(initial cervical dilatation: 2 to less than 3 cm)+4×(initial cervical dilatation: 3 to less than 4 cm)+10×(initial cervical dilatation: 4 cm or more)+2×(tobacco). The mean total score was 2.36±3.38 points (median 2.0, range 0–16). There was no significant difference in the predictive ability of the score compared with the predictive ability of the final multivariable model (AUC=0.76 compared with 0.75, P=.64).
The initial model to predict preterm birth at less than 37 weeks of gestation included all potential risk factors identified through unadjusted analyses (initial cervical dilatation, no prenatal care, tobacco use, obstetric history, and African American race). Performing serial AUC analyses after each step in backwards elimination confirmed that removing nonsignificant risk factors variables did not significantly reduce the predictive capability of the model. The final model for the secondary outcome included initial cervical dilatation, obstetric history, and tobacco use. This multivariable model was significantly more predictive of preterm birth at less than 37 weeks of gestation than initial cervical dilatation alone, with the univariable model having the greatest individual predictive ability (AUC=0.73 compared with 0.70, P=.006; Figure 2).
The adjusted odds ratios of each risk factor in the final multivariable model for preterm birth at less than 37 weeks of gestation along with the weighted estimate assigned to each risk factor are found in Table 4. The total score to predict preterm birth at less than 37 weeks of gestation can be expressed as: 0×(initial cervical dilatation: 0 to less than 2 cm)+2×(initial cervical dilatation: 2 to less than 3 cm)+4×(initial cervical dilatation: 3 to less than 4 cm)+15×(initial cervical dilatation: 4 cm or more)+ 0×(primiparous)+2×(previous preterm birth only)+ 1×(previous preterm birth and previous full-term birth)+1×(previous full-term birth only)+2×(tobacco). The mean total score was 3.39±4.50 points (median 2.0, range 0–19). There was no significant difference in the predictive ability of the score compared with the predictive ability of the final multivariable model (AUC=0.72 compared with 0.73, P=.27).
Table 5 demonstrates the test characteristics of different scores in predicting the risk of each adverse obstetric outcome. Using 2 or more as the cut-point for a positive compared with negative test result was associated with a 94.67% negative predictive value and 84.44% sensitivity in predicting delivery within 10 days, and an 81.54% negative predictive value and 78.92% sensitivity in predicting preterm birth at less than 37 weeks of gestation. Using this cut-point, there was a 55.89% prevalence of a positive test result for the primary outcome and a 66.04% prevalence of a positive test result for the secondary outcome. Test characteristics and 95% CIs associated with a total score of 2 or more generated using 1,000 bootstrap replications of the scoring system are presented in Table 6.
We have developed and internally validated two clinical prediction rules in a cohort of women symptomatic for preterm labor: one to predict delivery within 10 days of presentation with preterm labor and another to predict preterm birth at less than 37 weeks of gestation. The rule for the primary outcome included tobacco use during pregnancy, no prenatal care, and initial cervical dilatation. The rule for the secondary outcome included tobacco use during pregnancy, obstetric history, and initial cervical dilatation.
A negative test result for a particular outcome, obtained by not having any of the risk factors contained within the final model for that outcome, was associated with a strong ability to predict which patients would not experience delivery within 10 days (94.67%) and preterm birth at less than 37 weeks of gestation (81.54%). In fact, the test characteristics generated by our final score for delivery within 10 days were very similar to those of fetal fibronectin to predict delivery within 7 days when performed in patients symptomatic for preterm labor (positive predictive value 13%, negative predictive value 99.5%).26 This suggests that patients presenting with preterm labor could be risk-stratified for preterm birth without the use of expensive tools and advanced technology.
We are not the first investigators to attempt to create risk assessment systems for the prediction models for preterm birth based on demographic and clinical risk factors alone.27–33 However, previous investigators screened asymptomatic women and were unable to predict gestational age at preterm delivery. Furthermore, all studies except one were performed between 15 and 35 years ago.
There were several strengths to our study. Based on the results of our PubMed search described, we believe that we are the first group to specifically study women who presented with symptoms of preterm labor. Furthermore, whereas previous studies attempted to determine the odds of preterm birth, they did not attempt to predict what we believe to be a clinically more relevant outcome: the odds of relatively imminent delivery from the time of presentation with symptoms of preterm labor. Other strengths of our study include the fact that the data we used were obtained from enrolling a large number of high-risk symptomatic women in a prospective cohort study that limited misclassification bias. Furthermore, eligibility criteria for enrollment were determined before the start of the study by the primary and senior investigators and not by treating physician, which minimized potential enrollment biases.
Our study was not without limitations. First, although our rules were internally validated through use of the bootstrapping technique, our rules have not been externally validated in independent cohorts. Therefore, it is possible that these rules are specific to the population we investigated and may not be generalizeable to other populations with different demographics. Furthermore, our results may not be generalizeable for use in the outpatient or nontertiary care settings. Although we cannot be certain of the validity of the test characteristics of these models in the absence of external validation, internal validation with bootstrapping is a well-accepted technique.23,34,35
Another limitation is that, because of practice patterns at our institution, we do not have fetal fibronectin information for the patients in this cohort. We recognize that this information might have had a favorable effect on the model performance, particularly in terms of negative predictive value.
Other limitations are derived from the study design. For example, because we enrolled only those women who presented to the hospital (or who were directly counseled by their physicians to come to the hospital) because of their symptoms, our study does not include women who have symptoms concerning for preterm labor who chose not to present to the hospital. In addition, the use of symptomatic women prevents us from identifying those women who are at risk for preterm birth before the preterm labor process has started.
Patients with preterm premature rupture of membranes, a diagnosis on the spectrum of preterm labor, were included in this study. There are circumstances at our institution in which it is the standard of care to induce labor in patients with preterm premature rupture of membranes as previously described. However, there were only five patients in the entire cohort (0.86%) who required induction of labor within 10 days of presentation, and only four patients (0.69%) who required induction of labor at less than 37 weeks of gestation because of our institutional management of preterm premature rupture of membranes. Therefore, although it is possible that this inclusion may have affected our results, we do not believe that it biased our findings in any significant manner.
Although our clinical research coordinators were available to enroll patients 12 hours per day for 4 days per week and 8 hours per day on 2 additional days, our cohort was a convenience sample. However, it is unlikely that there were relevant systematic differences between patients who presented to the triage center for evaluation when clinical research coordinators were available compared with when clinical research coordinators were not available for enrollment. We acknowledge that, although our models demonstrated a statistically significant improvement in predicting each outcome over initial cervical dilatation alone, the incorporation of demographic variables appears to improve our clinical predictive ability only marginally. Finally, the test characteristics that we presented including the strong negative predictive value and sensitivity are based on our decision to use 2 or more as a cut-point for a positive compared with negative test result. Whereas the test characteristics would differ if we chose a different cut-point, we chose this value to maximize negative predictive value and therefore minimize the risk of mislabeling a patient who will experience preterm birth.
We believe that there may be situations in which providers either do not have access to or simply cannot use fetal fibronectin or cervical length screening. Therefore, the purpose of our study was to present alternative and purely clinical prediction tools with strong test characteristics that might be used by clinicians to corroborate their decision-making regarding the appropriate management of their patients.
The widespread application of this rule will require an understanding of the thresholds for treating compared with not treating preterm labor in diverse settings, with consideration of the potential risks and benefits of either strategy in a given population. However, before such widespread use, these rules must be validated in external populations and studied to determine whether their test characteristics can be significantly enhanced using additional and more detailed patient-specific data.
1. U.S. Department of Health and Human Services: Office of Disease Prevention and Health Promotion–Healthy People 2010. Nasnewsletter 2000;15:3.
2. Hamilton BE, Martin JA, Ventura SJ. Births: preliminary data for 2005. Natl Vital Stat Rep 2006;55:1–18.
3. Kurki T, Sivonen A, Renkonen OV, Savia E, Ylikorkala O. Bacterial vaginosis in early pregnancy and pregnancy outcome. Obstet Gynecol. 1992;80:173–7.
4. Rouse DJ, Hirtz DG, Thom E, Varner MW, Spong CY, Mercer BM, et al.. A randomized, controlled trial of magnesium sulfate for the prevention of cerebral palsy. N Engl J Med 2008;359:895–905.
5. Lyell DJ, Pullen K, Campbell L, Ching S, Druzin ML, Chitkara U, et al.. Magnesium sulfate compared with nifedipine for acute tocolysis of preterm labor: a randomized controlled trial. Obstet Gynecol 2007;110:61–7.
6. Antenatal corticosteroids revisited: repeat courses. NIH Consens Statement 2000;17:1–18.
7. Antenatal corticosteroid therapy for fetal maturation. ACOG Committee Opinion No. 475. American College of Obstetricians and Gynecologists. Obstet Gynecol 2011;117:422–4.
8. Effect of corticosteroids for fetal maturation on perinatal outcomes. NIH Consens Statement 1994;12:1–24.
9. Berkman ND, Thorp JM Jr, Hartmann KE, Lohr KN, Idicula AE, McPheeters M, et al.. Management of preterm labor. Evid Rep Technol Assess (Summ) 2000;18:1–6.
10. Berghella V, Ness A, Bega G, Berghella M. Cervical sonography in women with symptoms of preterm labor. Obstet Gynecol Clin North Am 2005;32:383–96.
11. Lockwood CJ, Senyei AE, Dische MR, Casal D, Shah KD, Thung SN, et al.. Fetal fibronectin in cervical and vaginal secretions as a predictor of preterm delivery. N Engl J Med 1991;325:669–74.
12. Iams JD, Goldenberg RL, Meis PJ, Mercer BM, Moawad A, Das A, et al.. The length of the cervix and the risk of spontaneous premature delivery. National Institute of Child Health and Human Development Maternal Fetal Medicine Unit Network. N Engl J Med 1996;334:567–72.
13. Roman AS, Koklanaris N, Paidas MJ, Mulholland J, Levitz M, Rebarber A. “Blind” vaginal fetal fibronectin as a predictor of spontaneous preterm delivery. Obstet Gynecol 2005;105:285–9.
14. Stafford IP, Garite TJ, Dildy GA, Colon-Lucach A, Williams CA, Bobritchi B, et al.. A comparison of speculum and nonspeculum collection of cervicovaginal specimens for fetal fibronectin testing. Am J Obstet Gynecol. 2008;199:131e1–4.
15. McKenna DS, Chung K, Iams JD. Effect of digital cervical examination on the expression of fetal fibronectin. J Reprod Med 1999;44:796–800.
16. Lumley J. The epidemiology of preterm birth. Baillieres Clin Obstet Gynaecol 1993;7:477–98.
17. Wen SW, Goldenberg RL, Cutter GR, Hoffman HJ, Cliver SP. Intrauterine growth retardation and preterm delivery: prenatal risk factors in an indigent population. Am J Obstet Gynecol 1990;162:213–8.
18. Ekwo EE, Gosselink CA, Moawad A. Unfavorable outcome in penultimate pregnancy and premature rupture of membranes in successive pregnancy. Obstet Gynecol 1992;80:166–72.
19. Cnattingius S, Granath F, Petersson G, Harlow BL. The influence of gestational age and smoking habits on the risk of subsequent preterm deliveries. N Engl J Med 1999;341:943–8.
20. Rosner B, Alumni and Friends Memorial Book Fund. Fundamentals of biostatistics. 6th ed. Belmont (CA): Thomson-Brooks/Cole; 2006.
21. Hosmer DW, Lemeshow S. Applied logistic regression. 2nd ed. New York (NY): Wiley; 2000.
22. Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983;148:839–43.
23. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996;15:361–87.
24. Moise A, Clement B, Ducimetiere P, Bourassa MG. Comparison of receiver operating curves derived from the same population: a bootstrapping approach. Comput Biomed Res 1985;18:125–31.
25. Kelsey JL, Thompson WD, Evans AS. Methods in observational epidemiology. New York (NY): Oxford University Press; 1986.
26. Peaceman AM, Andrews WW, Thorp JM, Cliver SP, Lukes A, Iams JD, et al.. Fetal fibronectin as a predictor of preterm birth in patients with symptoms: a multicenter trial. Am J Obstet Gynecol 1997;177:13–8.
27. Fedrick J. Antenatal identification of women at high risk of spontaneous pre-term birth. Br J Obstet Gynaecol 1976;83:351–4.
28. Creasy RK, Gummer BA, Liggins GC. System for predicting spontaneous preterm birth. Obstet Gynecol 1980;55:692–5.
29. Carr-Hill RA, Hall MH. The repetition of spontaneous preterm labour. Br J Obstet Gynaecol 1985;92:921–8.
30. Mercer BM, Goldenberg RL, Das A, Moawad AH, Iams JD, Meis PJ, et al.. The preterm prediction study: a clinical risk assessment system. Am J Obstet Gynecol 1996;174:1885–93; discussion 93–5.
31. Main DM, Gabbe SG, Richardson D, Strong S. Can preterm deliveries be prevented? Am J Obstet Gynecol 1985;151:892–8.
32. Tan H, Wen SW, Chen XK, Demissie K, Walker M. Early prediction of preterm birth for singleton, twin, and triplet pregnancies. Eur J Obstet Gynecol Reprod Biol 2007;131:132–7.
33. Lazar P, Dreyfus J, Papiernik E, Winisdoerfer G, Collin D, Gueguen S. [Prediction of prematurity as a preventive measure]. Arch Fr Pediatr 1978;35(Suppl 1):XIX–XXVI.
34. Steyerberg EW, Harrell FE Jr., Borsboom GJ, Eijkemans MJ, Vergouwe Y, Habbema JD. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 2001;54:774–81.
© 2012 by The American College of Obstetricians and Gynecologists. Published by Wolters Kluwer Health, Inc. All rights reserved.
35. Steyerberg EW, Bleeker SE, Moll HA, Grobbee DE, Moons KG. Internal and external validation of predictive models: a simulation study of bias and precision in small samples. J Clin Epidemiol. 2003;56:441–7.