Clark, Steven L. MD; Meyers, Janet A. RN; Milton, Celeste G. MPH, RN; Frye, Donna R. RN, BSN; Horner, Stephen RN, BSN; Baker, Allison MS; Perlin, Jonathan B. MD, PhD
OBJECTIVE: To evaluate whether current Joint Commission (JC) exclusion criteria for measure PC-01, “Elective Delivery” before 39 weeks of gestation, accurately identify valid, codeable indications for planned early-term delivery.
METHODS: We performed a review and critical analysis of all cases recorded as noncompliant for the measure in a large health care system during the second half of 2012.
RESULTS: During the study period, of 107,145 total deliveries, 205 cases were reported as noncompliant with PC-01. Ten percent of compliance fallouts (ie, cases coded as noncompliant) resulted from valid indications for delivery identifiable by International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) coding not included on the JC exclusion list; these were primarily unusual or extreme variations of these conditions. Twenty-five percent of fallouts represented valid indications not represented by an ICD-9-CM code. Eight percent of cases were reported as fallouts as a result of imprecise physician charting; only 2% represented chart abstraction errors. Fifty-five percent of cases involved stated indications for early-term delivery not generally recognized as such by the medical community. Compliance rates of 98% are achievable across a large population using the current ICD-9-CM–based metric for compliance assessment used by the JC (PC-01). The current exclusion list does not appear to be amenable to further improvement by inclusion of more or different ICD-9-CM codes. However, given the low volumes generated using the current PC-01 denominator definition, approximately 60% of facilities would have compliance rates below a 95% benchmark with even a single justified outlier if analyzed on a quarterly basis.
CONCLUSION: Our data validate the current JC exclusion criteria for this measure, which identify the vast majority of valid indications for early-term delivery used by obstetrician–gynecologists and identifiable with ICD-9-CM codes.
LEVEL OF EVIDENCE: III
Since the initial descriptions of increased neonatal morbidity associated with elective early-term delivery, widespread efforts have been made to reduce or eliminate this practice and the morbidity associated with it.1,2 After adoption by the National Quality Forum, a quality metric describing compliance with efforts to reduce elective early-term delivery was developed by The Joint Commission (JC; PC-01); this metric has also been adopted by the Centers for Medicare & Medicaid Services and public reporting agencies such as Leapfrog and is coming under increased scrutiny by third-party payers and even politicians.3–9
Given the increasing importance of this JC metric, we undertook a critical evaluation of their list of acceptable criteria for indicated early-term delivery in a large obstetric population. We also sought to determine whether currently reported national compliance goals are achievable in most hospitals and if these “benchmarks” are appropriate to use for accountability, payment determination (or both) by commercial and governmental payers given the critical effect of rare, clinically justifiable outliers on sample sizes commonly observed in U.S. facilities. Such an analysis is critical if this metric is to be appropriately and effectively used.10
MATERIALS AND METHODS
Affiliated hospitals of the Hospital Corporation of America deliver just more than 200,000 neonates annually in 108 hospitals in 22 states. Since the adoption of the elective early-term delivery metric by the JC, these facilities have collected and reported internally quarterly compliance data in accordance with sampling and reporting criteria established by the JC using an ORYX vendor that has met the criteria for inclusion in the accreditation process and is included on the JC's list of accepted ORYX vendors.11 The process of data extraction for this measure is governed by an exacting JC algorithm carried out in a uniform manner by coders in each institution. This process is detailed in a flowchart on the JC's web site at https://manual.jointcommission.org/releases/TJC2013A/MIF0166.html. For submission of core measures to the JC, all hospitals are required to use a vendor that has met criteria for inclusion in the accreditation process and is included in the JC's list of acceptable ORYX vendors. These organizations must meet a list of 21 attributes of conformance addressing data quality and completeness and are audited by the JC for compliance. As an additional step in ensuring data quality and completeness, the Hospital Corporation of America conducts independent interrater reliability testing on a quarterly basis. A random sample of records is selected, the previous abstraction results blinded, and the records are reabstracted by a different reviewer. Agreement rates between the two abstractions as well as category assignment (numerator, denominator, and excluded cases) are examined. Prompt corrective actions are taken in facilities that do not meet established agreement thresholds. The two quarters of data reported in this article were subjected to this exacting data quality control process to ensure the accuracy of data and proper application of the data extraction algorithm. In addition, each outlier data point reported here was reviewed both by a nursing supervisor at the facility level and by two authors of this article at the system level, a board-certified maternal–fetal medicine specialist and an experienced labor and delivery nurse who oversees obstetric nursing quality for our system. Any questions regarding clinical details or classification were resolved by the principle author and local coders or clinicians before inclusion in this reported data set.
For the last two quarters of 2012, we asked each facility to report to us not only compliance rates, but details on the stated indications for all cases reported as noncompliant. These “indications” for delivery were then tabulated and analyzed both in terms of actual medical validity and with respect to the availability of an International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) code to describe the indication to determine the completeness of the current JC exclusion criteria list. Listed indications categorized in Table 1 as invalid and not generally accepted by the medical community as justified were identified as such by two of the author clinicians (S.L.C., D.R.F.). The author associated with the JC (C.G.M.) was instrumental in leading the development of JC measure PC-01 and the exclusion criteria described in this report; however, to avoid any conflict of interest, all data collection and analysis of data relating to completeness of the current JC list of exclusion criteria was conducted and finalized by authors not affiliated with the JC.
Based on these data, we also examined the population size necessary to draw statistically valid inferences regarding comparative compliance with various potential compliance benchmarks using a dichotomous independent prospective two-proportioned uncorrected χ2 test assuming equal numbers of cases and controls. All information was deidentified before submission, and the project was directed at quality improvement. As such, it was exempt from institutional review board approval.
In 2012, 107,145 deliveries occurred at Hospital Corporation of America-affiliated institutions in the United States from July 1 to December 31. Using established JC sampling and reporting methodology, 205 inductions of labor or planned cesarean deliveries between 37 weeks 0 days and 38 weeks 6 days of gestation were identified locally as not meeting the JC's ICD-9-CM code criteria for indicated delivery and were reported as fallouts. The mean combined compliance rate for these two quarters was 96.5% (2 standard deviations=86.2–100%) with rates of 96.0% and 97.1% reported individually for quarters 3 and 4, respectively. Almost two thirds of facilities in both quarters had quarterly compliance rates of 100%. Distribution of individual facility reports over two quarters is detailed in Figure 1.
PC-01: Elective Deli...Image Tools
Closer analysis of the stated reasons for delivery in noncompliant cases allowed categorization into five groups (Table 1). Group 1 included those patients reported as fallouts (ie, noncompliant cases) as a result of errors in chart abstraction. Such errors were negligible (2% of all reported fallouts). Group 2 included patients who had indications for delivery recognized by the JC but in whom physician documentation was insufficient to allow identification of an accepted exclusion. Examples were women delivered for “low amniotic fluid” rather than for the approved and synonymous standard term “oligohydramnios” or a woman delivered for the highly specific diagnosis “Eisenmenger syndrome” rather than one of the more generic and JC-recognized indications for delivery before 39 weeks of gestation involving cardiovascular disease. Such cases were also uncommon, making up only 8% of reported fallouts. Group 3 (10%) included those women whose indication for delivery was both valid and had a recognized ICD-9-CM code but was not on the JC list of approved exceptions. These consisted primarily of deliveries performed for indications not generally considered to justify early-term delivery but which as a result of the unusual nature or severity of the condition appeared to be appropriate in the specific case in question. An example would be a woman with epilepsy (generally not an indication for early-term delivery) delivered at 38 weeks of gestation because of recurrent grand mal seizures abruptly increasing in frequency in the last weeks of pregnancy. This group would also include occasional very rare medical conditions not on the JC exclusion list. Group 4 (25%) included women with valid indications for early-term delivery, which do not have an assigned ICD-9-CM code. A multiparous woman at 38 3/7 weeks of gestation with a history of precipitous delivery who was found to be 5 cm dilated and who lived a significant distance from the hospital would be one such example. Group 5 (55%) included women whose stated indication for early-term delivery was not a generally recognized medical or obstetric indication for early-term delivery. Examples would be “impending macrosomia” or nonspecific “abdominal discomfort.” The distribution of such true outliers was highly skewed; 57% originated in 7% of facilities.
Recent interest in early-term delivery began with our demonstration in 2009 of increased morbidity for neonates delivered both vaginally and by repeat or primary cesarean delivery at 37 and 38 weeks of gestation compared with 39 weeks of gestation and beyond.1 The findings of Tita et al confirmed these observations in neonates delivering by repeat cesarean.2 Several subsequent studies have confirmed these initial observations as well as the effect of focused efforts to reduce elective early-term delivery.2,12–16
In November 2007, the JC's Board of Commissioners recommended retiring the existing Pregnancy and Related Conditions core measure set. The Board also made a further recommendation to replace these measures with an expanded set of measures based on current scientific evidence.3 The new set of measures known as the perinatal care core measure set was then adapted from the national perinatal care measures endorsed in October 2008 by the National Quality Forum.4,5
A Perinatal Care Technical Advisory Panel appointed by the JC in December 2008 selected five measures from the 17 National Quality Forum-endorsed perinatal care measures. Among the five measures recommended for inclusion was PC-01, which addressed elective early-term delivery.3
This and other perinatal care measures were fully specified in a standardized manner similar to that of the JC's existing core measure sets. Numerator and denominator populations were defined, a data dictionary was built, measure calculation algorithms were developed, and code tables were constructed to identify included and excluded populations. Deliveries performed between 37 weeks 0 days and 38 weeks 6 days of gestation not identified as exclusions in the JC measure are reported as fallouts and measured against compliance rate. During this time, JC staff worked closely with the original measure developer (the Hospital Corporation of America) and other perinatal care experts to determine appropriate exclusions for PC-01 based on field testing experience and the American College of Obstetricians and Gynecologists clinical management guidelines for induction of labor.
We subsequently compared three approaches to reduction of early-term delivery and demonstrated that a “hard stop” approach involving predelivery evaluation of the indication for planned early-term delivery by the nursing staff, with referral, when necessary to the medical chain of command, was highly effective in reducing perinatal morbidity, newborn intensive care admission, and cost in these patients.17 Since that time, our policy has been to encourage universal adoption of a “hard stop” approach in all Hospital Corporation of America-affiliated facilities. Even with the extensive focus on the issue of elective early-term delivery described here, and the achievement of a compliance rate of 96.4% during the study period, we found that approximately half of the persistent fallouts could be eliminated and a 98% compliance rate achieved in a large population with universal evidence-based physician decision-making (Table 1). Our data also suggest that modification of the current list of codes would not significantly improve capture of justified measure exclusions; virtually all medically justified indications for early-term delivery not included in the current JC exclusion list either have no ICD-9-CM code or represent extreme and unusual variations of conditions that are associated with an ICD-9-CM code but which only rarely justify early-term delivery. Information available to us at this time suggests that this will not change with the availability of anticipated International Classification of Diseases, 10th Revision codes. Although these data suggest that 98% compliance would probably represent a theoretical threshold for an ICD-9-CM–based coding analysis that cannot be consistently exceeded in a large population in the long term without compromising patient care, in reality, most individual facilities should and can achieve 100% compliance for most quarters. Indeed in the first quarter of 2013, a system-wide compliance rate (108 hospitals) of 98% was achieved through continued efforts to eliminate unwarranted early-term delivery. Changing physician practice in this area is clearly not a short-term project, particularly in systems such as ours largely consisting of nonemployed physicians, an absence of hospital-controlled insurance panels, and independent medical staffs.17,18
Of concern is the possibility of making invalid comparisons between groups based on an insufficient sample size resulting from the small proportion of women who deliver in the 13-day range defined by the JC metric denominator. Although two thirds of our hospitals achieved a 100% compliance rate in each quarter, a single valid fallout that is missed using ICD-9-CM criteria using quarterly data would yield a compliance rate less than 95% in 65% of facilities (Fig. 1; also see the Appendix online at http://links.lww.com/AOG/A453). Indeed, a denominator sample size of 194 in each group would be necessary to conclude with sufficient power that a 97% compliance rate is statistically different than a 90% compliance rate assuming an 80% power to detect a difference (α=0.05). Such discrimination is not possible on a quarterly basis using standard statistical validity testing even with an annual delivery volume of 8,000 patients given the current JC denominator definition for this metric. Thus, performance comparisons based on a target compliance rate of 98% and quarterly data will fail standard statistical validity tests for virtually all facilities in the United States. If such comparisons are to be performed on relatively rare events, more sophisticated methods such as Poisson regression would be necessary to avoid invalid conclusions regarding differences in compliance rates. Alternately, confidence intervals (CIs) should be calculated based on denominator size and used in any comparative performance analysis.
This statistical analysis does not in any way diminish the validity or importance of JC measure PC-01 nor does it suggest the need to revise target benchmarks. Rather, our data emphasize the importance and feasibility of 100% compliance with a program aimed at the elimination of elective early-term deliveries, although a practical, ICD-9-CM code–based system for compliance assessment may not capture 1–2% of indicated deliveries. However, given the complexity of this issue, a 98% capture rate in a coding-based system is remarkable. These data also serve to emphasize that this metric is of primary importance in identifying good rather than bad performance over a single quarter. A facility meeting the desired 100% compliance rate for any quarter has performed well during that period; on the other hand, compliance rates below this level require more detailed clinical analysis of the indications associated with reported fallouts before conclusions regarding possible poor performance can be reached. These statistical observations suggest the need for great caution in interpreting, publishing, or acting on comparative data based on an insufficient denominator. This may be of particular importance for public reporting groups and third-party payers; it would be essential for such parties to base performance thresholds on Poisson regression or CIs specific to the population size in question to avoid penalizing facilities or health care practitioners based on statistically invalid methodology.
The strengths of this study derive primarily from the large numbers of coding fallouts identified in our large delivery population. Given the relatively small measure denominators resulting from the JC measure specifications, this degree of analysis would not be possible in any individual facility and would be very difficult in health care systems with fewer deliveries than the Hospital Corporation of America. Finally, intense focus put on compliance with this measure within our system in the past several years allows conclusions regarding the eventual practical limits on compliance for hospitals and hospital systems in general. The principle limitation of this study involved the need for a clinical judgment, made by two authors, regarding whether a stated indication for delivery was, or was not, generally accepted as such by the medical community and supported by evidence-based medicine. This degree of subjectivity could affect the precise distribution of cases between categories 4 and 5 (Table 1) but would not significantly affect our major conclusions regarding the validity of the current JC measure exclusion criteria or our observations regarding the statistical validity of comparative conclusions based on small denominators.
In conclusion, we believe the JC early elective delivery metric to be one of the most important of all recently developed performance measures in terms of its demonstrated effect on practice patterns as well as reduction in morbidity and cost. Our data validate the performance of the current JC metric and its associated exclusion criteria. Finally, we caution against statistically invalid comparisons and benchmarking using this metric, which could discredit and diminish the acceptance of this vital performance measure.
1. Clark SL, Miller DD, Belfort MA, Dildy GA, Frye DK, Meyers JA. Neonatal and maternal outcomes associated with elective term delivery. Am J Obstet Gynecol 2009;200:156.e1–4.
2. Tita AT, Landon MB, Spong CY, Lai Y, Leveno KJ, Varner MW, et al.. Timing of elective repeat cesarean delivery at term and neonatal outcomes. N Engl J Med 2009;360:111–20.
5. Milton CG. The joint commission’s perinatal care measure set overview. Breastfeed Med 2010;5:257–8.
10. Clark SL, Meyers JA, Perlin JB. Oversight of elective early term deliveries: avoiding unintended consequences. Am J Obstet Gynecol 2012;206:387–9.
12. Oshiro BT, Henry E, Wilson J, Branch DW, Varner MW; Women and Newborn Clinical Integration Program. Decreasing elective deliveries before 39 weeks of gestation in an integrated health care system. Obstet Gynecol 2009;113:804–11.
13. Donovan EF, Lannon C, Bailit J, Rose B, Iams JD, Byczkowski T; Ohio Perinatal Quality Collaborative Writing Committee. A statewide initiative to reduce inappropriate scheduled births at 36 0/7 to 38 6/7 weeks' gestation. Am J Obstet Gynecol 2010;202:243.e1–8.
14. Reddy UM, Bettegowda VR, Dias T, Yamada-Kushnir T, Ko CW, Willinger M. Term pregnancy: a period of heterogeneous risk for infant mortality. Obstet Gynecol 2011;117:1279–87.
15. Kamath BD, Morcotte MP, DeFranco EA. Neonatal morbidity after documented fetal lung maturity in late preterm and early term infants. Am J Obstet Gynecol 2011;204:518.e1–8.
16. Oshiro BT, Kowalewski L, Sappenfield W, Alter CC, Bettegowda VR, Russell R, et al.. A multistate quality improvement program to decrease elective deliveries before 39 weeks. Obstet Gynecol 2013;121:1025–31.
17. Clark SL, Frye DR, Meyers JA, Belfort MA, Dildy GA, Kofford S, et al.. Reduction in elective delivery prior to 39 weeks gestation—comparative effectiveness of 3 approaches to change and the impact on neonatal outcome and stillbirth. Am J Obstet Gynecol 2010;203:449.e1–6.
18. Clark SL, Meyers JA, Frye DK, Perlin JA. Patient safety in obstetrics: the Hospital Corporation of America experience. Am J Obstet Gynecol 2011;204:283–7.
© 2014 by The American College of Obstetricians and Gynecologists.