Effectiveness of a Multivariate Index Assay in the Preoperative Assessment of Ovarian Tumors

Abstract

OBJECTIVE: 

To compare the effectiveness of physician assessment with a new multivariate index assay in identifying high-risk ovarian tumors.

METHODS: 

The multivariate index assay was evaluated in women scheduled for surgery for an ovarian tumor in a prospective, multi-institutional trial involving 27 primary- care and specialty sites throughout the United States. Preoperative serum was collected, and results for the multivariate index assay, physician assessment, and CA 125 were correlated with surgical pathology. Physician assessment was documented by each physician before surgery. CA 125 cutoffs were chosen in accordance with the referral guidelines of the American College of Obstetricians and Gynecologists.

RESULTS: 

The study enrolled 590 women, with 524 evaluable for the multivariate index assay and CA 125, and 516 for physician assessment. Fifty-three percent were enrolled by nongynecologic oncologists. There were 161 malignancies and 363 benign ovarian tumors. Physician assessment plus the multivariate index assay correctly identified malignancies missed by physician assessment in 70% of nongynecologic oncologists, and 95% of gynecologic oncologists. The multivariate index assay also detected 76% of malignancies missed by CA 125. Physician assessment plus the multivariate index assay identified 86% of malignancies missed by CA 125, including all advanced cancers. The performance of the multivariate index assay was consistent in early- and late-stage cancers.

CONCLUSION: 

The multivariate index assay demonstrated higher sensitivity and lower specificity compared with physician assessment and CA 125 in detecting ovarian malignancies.

LEVEL OF EVIDENCE: 

III

In Brief

The multivariate index assay is a preoperative diagnostic assay with high sensitivity for identifying all stages of ovarian malignancy, facilitating decisions about referral to a gynecologic oncologist.

National and international guidelines support the participation of a gynecologic oncologist in the care of women with ovarian cancer.^1–7 A meta-analysis of 18 studies found that early involvement of a gynecologic oncologist improved outcomes, concluding: 1) early-stage disease is more likely to have surgical staging, 2) advanced disease is more likely to receive optimal cytoreductive surgery, and 3) advanced disease has an improved median and overall 5-year survival.⁸ Yet only one third of women with ovarian cancer are referred to a gynecologic oncologist for primary surgery.^9,10

To decide who should surgically remove an ovarian tumor, physicians interpret patient history, examination, imaging studies, and laboratory tests. Examination is an unreliable method to detect ovarian tumors.^11,12 Pelvic ultrasonography is useful in establishing the risk of tumors based on size and morphology,^13,14 but it cannot differentiate benign from malignant consistently, particularly when ovarian size is normal. CA 125 is indicated for monitoring ovarian cancer but not for screening or preoperative assessment. CA 125 has low sensitivity (50%) in early-stage cancers^15,16 and low specificity in both premenopausal and postmenopausal women.¹⁶ The American College of Obstetricians and Gynecologists (the College) referral criteria⁴ and the Risk of Malignancy Index^17,18 use CA 125, but neither is universally accepted in the United States.

Multivariate index assay is a new multivariate biomarker assay used with physician assessment to determine whether an ovarian tumor warrants referral for surgery. The objective was to compare the effectiveness of physician assessment with a multivariate index assay in identifying high risk ovarian tumors.

MATERIALS AND METHODS

Patients were enrolled from 27 sites throughout the United States, including women's health clinics, obstetrics and gynecology groups, gynecologic oncology practices, community and university hospitals, and health maintenance organizations (see the Appendix, available online at https://links.lww.com/AOG/A240). Institutional review board approval was obtained from each site. Volunteer participants represent a consecutive series of patients who met inclusion criteria from each participating institution. The criteria for inclusion were: female, age 18 years or older, a documented ovarian tumor with planned surgical intervention within 3 months of imaging, agreeable to phlebotomy, and signed informed consent. Before enrollment, all ovarian tumors were confirmed with an imaging study (computed tomography scan, ultrasonography, or magnetic resonance image). The exclusion criteria were age younger than 18, no planned surgical intervention, declined phlebotomy, or had a malignancy diagnosis in the previous 5 years with the exception of a nonmelanoma skin cancer. Menopause was defined as the absence of menses for at least 12 months or age 50 or older when the woman was unsure of her menopausal status. All demographic, clinical, and pathologic information was faxed to Applied Clinical Intelligence for analysis.

Thirty to 50 mL of preoperative venous blood was collected from each patient and placed in BD plastic vacutainer tubes with clot activators and centrifuged after sitting at 18–25°C for a minimum of 1 hour and maximum of 6 hours postphlebotomy. Specimens for each patient were pooled before aliquoting, stored at -65°C to -85°C, and shipped frozen to PrecisionMed International for storage. Biomarker measurements were performed by Quest Diagnostics, Inc., and blinded validation testing was done at Johns Hopkins Medical Institutions and Specialty Laboratories.

The multivariate index assay test incorporates CA 125-II, transferrin, transthyretin (prealbumin), apolipoprotein AI, and beta 2 microglobulin. Many of these individual biomarkers,^15,16,19 and the rationale for the development of the multivariate index assay test,²⁰ have been previously described. The multivariate index assay algorithm was derived and validated from two independent serum training sets. The clinically relevant cutoffs were selected to maximize the index over its individual components while maintaining a high level of sensitivity and negative predictive value. The proprietary OvaCalc software combines the values for each assay, and uses the multivariate index assay algorithm to generate an ovarian malignancy risk index score for each. The numeric result ranges from 0.0 and 10.0, with the following clinical report:

• Premenopausal
• Low probability of malignancy (multivariate index assay less than 5.0)
• High probability of malignancy (multivariate index assay 5.0 or higher)

• Postmenopausal
• Low probability of malignancy (multivariate index assay less than 4.4)
• High probability of malignancy (multivariate index assay 4.4 or higher)

CA 125-II was measured on the Elecsys 2010 (Roche Diagnostics) and the other four markers were measured on the BN II System (Siemens Healthcare Diagnostics).

For each patient, a standard CA 125-II assay was performed. The same assay value was used in the analysis for both CA 125 and the multivariate index assay algorithm. The CA 125 clinical cutoff values were selected in accordance with the published College referral criteria^4,21 as more than 200 units/mL for premenopausal women and more than 35 units/mL for postmenopausal women. We also evaluated the modified College criteria proposed by Dearking as more than 67 units/mL for premenopausal women.²²

Before surgery, clinicians were required to document the results of the physical examination, family history, imaging, laboratory tests (including CA 125 but not the multivariate index assay), and physician assessment. The majority (80%) of physicians ordered CA 125 as part of their assessment. The enrolling physician formally declared their preoperative assessment by answering the question, “based on all available clinical information, is the physician of the opinion that this is a malignant ovarian tumor? (yes or no).” Physicians were allowed to use any algorithm they desired to determine their answer, but where not required to explain how they arrived at their prediction. A physician assessment was recorded for 516 of 524 patients.

Statistical analysis was stratified based on menopausal status, physician specialty, stage of malignancy, and surgical pathology. Concordances between multivariate index assay results of high or low probability of malignancy and pathologic findings were assessed using McNemar's χ² test. Clinically relevant criteria such as sensitivity, specificity, positive predictive value, negative predictive value, and likelihood ratios were calculated to estimate the performance of multivariate index assay, CA 125, preoperative assessment alone, and multivariate index assay with preoperative assessment. Ninety-five percent confidence intervals were constructed where appropriate, and those for the area under the receiver operating characteristic curve were calculated using a bootstrap procedure. Statistical analysis was performed with SAS 9.1 (SAS Institute Inc).

RESULTS

From 2007 through 2008, 590 women were enrolled in the study, of which 524 were evaluable for both multivariate index assay and CA 125, and 516 for physician assessment. All patients had an imaging study verifying an ovarian mass. Women were excluded from analysis if surgery was either not performed (27) or delayed more than 3 months (three), pathology report was not available (26), blood specimen was unusable (nine), physician assessment was not available (eight), or imaging study did not confirm an adnexal tumor (one). The clinical characteristics of the study group are summarized in Table 1. More than half of the patients (53%) were enrolled by nongynecologic oncologist. Fewer than 10% of patients had clinical evidence of advanced disease at study enrollment (ascites 9.7%, metastases 4.4%). Final pathology identified 161 malignancies and 363 benign tumors. There were 151 ovarian malignancies (29%), including: 94 epithelial ovarian cancers, 11 nonepithelial ovarian malignancies, 28 ovarian borderline tumors, and 18 malignancies metastatic to the ovary. Of the 10 patients with nonovarian malignancies, one had a synchronous borderline ovarian tumor and an endometrial cancer and nine with documented adnexal tumors on imaging studies had pelvic malignancy but normal ovarian histology (uterine 4, retroperitoneal 2, fallopian tube 1, small bowel 1, other 1).

Table 1: Summary of Clinical Characteristics for All Eligible Patients

The performances of physician assessment, multivariate index assay and CA 125 for predicting ovarian malignancy are summarized in Tables 2 and 3. The multivariate index assay improved the sensitivity of physician assessment regardless of gynecologic specialty. The multivariate index assay alone and the multivariate index assay with physician assessment also had the highest sensitivity and negative predictive value, but the lowest specificity and positive predictive value when compared with CA 125 and physician assessment alone. Using McNemar's test, the sensitivity of the multivariate index assay was statistically superior to physician assessment (χ²=20.6, P<.001), CA 125 with cutoff of 200 units/mL (χ²=38.0, P<.001), and CA 125 with cutoff of 67 units/mL (χ²=25.0, P<.001). Receiver operating characteristic curve analysis of the multivariate index assay is shown in Figure 1. The sensitivity of the multivariate index assay test by histologic subtype is: epithelial ovarian cancer 99% (93/94), nonepithelial ovarian cancer 82% (9/11), borderline ovarian tumor 75% (21/28), and metastases to ovary 94% (17/18).

Table 2: Performance of Physician Assessment Alone and With the Multivariate Index Assay in Predicting Ovarian Malignancy

Table 3: Test Performance in Predicting Ovarian Malignancy

Fig. 1.: Receiver operating characteristic curve analysis of a multivariate index assay in the risk of malignancy assessment for ovarian tumors. The area under the curve was 0.80 (95% confidence interval [CI] 0.73–0.88) for premenopausal women and 0.82 (95% CI 0.77–0.87) for postmenopausal women.Ueland. Multivariate Index Assay to Assess Ovarian Tumors. Obstet Gynecol 2011.

When compared with physician assessment, physician assessment plus the multivariate index assay demonstrated a significant improvement in sensitivity and negative predictive value which was independent of physician specialty (Table 2). Adding the multivariate index assay to physician assessment correctly identified 70% (14/20) of malignancies missed by nongynecologic oncologists, and 95% (19/20) of malignancies missed by gynecologic oncologists. The multivariate index assay had higher sensitivity than either CA 125 or physician assessment (Table 3); however, the addition of the multivariate index assay to physician assessment decreased both specificity and positive predictive value (Tables 2 and 3). In this trial, there were 363 benign ovarian tumors (355 with physician assessment) and 161 malignancies. Of all patients referred for surgery to the gynecologic oncologist, 38% were found to have a malignancy on surgical pathology. There were 261 benign tumors referred to the gynecologic oncologist for surgery, a benign tumor referral rate of 72% (261/363). As documented by physician assessment, the generalist physician predicted that 183 tumors were benign before surgery, and still referred 83 (45%) to the gynecologic oncologist for surgery.

The multivariate index assay test maintained high sensitivity regardless of cancer stage, with 90% sensitivity for stage I cancers, and 100% sensitivity for stages II through IV (Table 4). The sensitivity of the multivariate index assay in early-stage disease (stage I and II) is 94% compared with 61% for CA 125. Moreover, for the cancers missed by physician assessment alone, 70% of the primary ovarian cancers are early-stage (I or II), and 58% had a normal CA 125 value. For early-stage epithelial ovarian cancer, the multivariate index assay sensitivity was 98% compared with 68% for CA 125 (Table 5). Both tests effectively identified late-stage disease (stage III and IV). The sensitivity of the multivariate index assay for all stages of epithelial ovarian cancer and all primary ovarian malignancies was 99% and 97%, respectively.

Table 4: Sensitivity by Stage for Multivariate Index Assay and CA 125-II in Identifying Primary Ovarian Malignancy

Table 5: Sensitivity by Stage for Multivariate Index Assay and CA 125-II in Identifying Epithelial Ovarian Cancer

In premenopausal women with early-stage disease (Tables 4 and 5), the multivariate index assay was more sensitive than CA 125 for all primary ovarian malignancies (82% compared with 29%), and for epithelial ovarian cancers (93% compared with 36%). The multivariate index assay was also more sensitive than CA 125 in postmenopausal women with early-stage disease (Tables 4 and 5). Both tests were effective at detecting late-stage disease, regardless of menopausal status. In this study, 21 of 27 premenopausal patients with ovarian malignancy had positive multivariate index assay tests but indeterminate CA 125 scores (200 units/mL or less), including 9 of the 12 with stage I or II disease. In addition, 17 of 23 postmenopausal women had positive multivariate index assay tests and negative CA 125 scores (35 units/mL or less). Overall, 76% (38/50) of patients with ovarian malignancy had positive multivariate index assay tests despite indeterminate CA 125 results. With the addition of the multivariate index assay, the sensitivity of physician assessment for any ovarian malignancy improved from 60% to 89% (premenopausal) and 81 to 98% (postmenopausal). As expected, in premenopausal women, the negative predictive value increased slightly (90% compared with 94%) and the specificity and positive predictive value decreased (83% compared with 40%; 46% compared with 26%, respectively). Similar trends were noted in postmenopausal women. The multivariate index assay clinical performance for detecting malignancy at varying cutoff levels is displayed for premenopausal and postmenopausal women in Table 6.

Table 6: Clinical Performance of the Multivariate Index Assay With Varying Cutoff by Menopausal Status

DISCUSSION

The multivariate index assay is a diagnostic test that complements the physician's preoperative assessment; it is not a screening test for ovarian cancer. Although there are many published guidelines for the evaluation of an ovarian tumor,^{4,5,14,17,18,23–26} none are universally accepted. It is apparent from physician decisions in this trial that there is considerable autonomy in preoperative evaluations, with few following explicit algorithms. The College guidelines permit the physician to choose an imaging modality which is combined with CA 125, patient history, and physical examination. The Risk of Malignancy Index relies exclusively on CA 125, ultrasonography, and menopausal status. Other biomarker algorithms have been proposed,²⁶ but conclusions are premature because they have not been studied in populations in which the test is indicated for use. The dispute with the College criteria is that, although the guidelines are successful in predicting advanced ovarian cancer,^21,22 they do not perform well in identifying early-stage disease and particularly in premenopausal women, presumably because CA 125 is less sensitive in these populations.²² Premenopausal women are particularly challenging because they produce the majority of benign cystic tumors and comprise 20% of all ovarian cancers.^27,28 Surprisingly, only 10% of women with presumptive early ovarian cancer receive the recommended staging and treatment,²⁹ which is important because their survival is incrementally better.³⁰ The CA 125 value has a considerable influence on the Risk of Malignancy Index and the College criteria. The College guidelines use CA 125 cutoff values for premenopausal (more than 200 units/mL) and postmenopausal (more than 35 units/mL) women, whereas the Risk of Malignancy Index is linked arithmetically to CA 125 because the calculated index score is a product of the actual CA 125 value.^18,19 The performance of the Risk of Malignancy Index has not been reported for early-stage disease or premenopausal women. Identifying malignancies for referral in these challenging cohorts is important for comprehensive staging, cytoreduction, appropriate chemotherapy, access to clinical trials, and better outcomes.^8–10

To be clinically useful, a preoperative cancer referral test must have high sensitivity. The four additional biomarkers in the multivariate index assay test improve the sensitivity of detecting malignancy when compared with CA 125. Although in clinical practice neither biomarker test is used independently, the multivariate index assay and CA 125 are individually compared to better understand the behavior of each. Considering all patients with pelvic malignancies, the multivariate index assay had a significantly higher sensitivity compared with CA 125. Likewise, the negative predictive value of the multivariate index assay was higher compared with CA 125, regardless of chosen cutoff. These trends were consistent for early- and late-stage cancers, premenopausal and postmenopausal women. When compared with physician assessment alone (which includes the use of CA 125), physician assessment with the multivariate index assay showed similar improvements in sensitivity and negative predictive value. In all, the multivariate index assay identified 76% (38/50) of malignancies missed by CA 125 (more than 200 units/mL), and physician assessment plus the multivariate index assay detected 82% (33/40) malignancies missed by physician assessment alone, and 86% (43/50) of malignancies missed by CA 125.

If used appropriately, the multivariate index assay should not result in additional surgery as it is approved for use in women already scheduled for surgery. It is not known precisely how the multivariate index assay will affect patient referrals. The addition of the multivariate index assay to physician assessment decreased specificity and positive predictive value, a consequence of more false-positive results. This implies that women with nonmalignant tumors may be identified for referral to a gynecologic oncologist. Admittedly, the decision to refer a patient is complex, influenced by many medical and nonmedical concerns. In actual practice, lower specificity may not necessarily result in more benign referrals. Twelve to 40% of women referred to a gynecologic oncologist have a confirmed ovarian malignancy at surgery,^14,2,22 meaning more than 60% are referred for benign disease. In this multivariate index assay trial, the specificity of physician assessment alone (nongynecologic oncologists) was 83%, yet the number of nonmalignant tumors referred to the gynecologic oncologist was still high. Surprisingly, 72% of all benign ovarian tumors were referred to a gynecologic oncologist for surgery, including 45% of patients referred despite a negative physician assessment. So the calculated specificity and predictive values of physician assessment do not correlate with observed patient referrals. In patients with a nonmalignant preoperative assessment, the higher negative predictive value of the multivariate index assay may be sufficiently reassuring to discourage referral. Ultimately, this nonrandomized trial was not designed to evaluate the effect of the multivariate index assay on benign tumor referrals.

The strengths of this study include independent biomarker validation, multi-center prospective data collection, and inclusion of all ovarian tumor types for relevance to general gynecologic practice. Also, the majority of patients in the multivariate index assay trial were enrolled at primary care sites where the test is intended for use. A possible study limitation is the lack of a uniform preoperative evaluation for comparison; however, physician assessment was an intentional study design. Preoperative assessment is the most realistic clinical surrogate given that other referral algorithms are not uniformly accepted in the United States. All women enrolled in this trial were scheduled for surgery for an ovarian tumor. This is a cohort with increased cancer prevalence compared with those in whom surgery is not planned. A population with lower cancer prevalence would decrease the test's positive predictive value and increase the negative predictive value. A multivariate index assay is not required if the preoperative evaluation without the multivariate index assay is highly suggestive of malignancy, and the multivariate index assay is not approved for serial surveillance of ovarian tumors without surgery. Hopefully, earlier referral of patients with ovarian cancer will improve survival and reduce the number of required re-operations. The overall value and cost-effectiveness of the multivariate index assay will be determined by future health studies evaluating both economic and outcome measures.