Evans, Mark I. MD; Krantz, David A. MA; Hallahan, Terrence W. PhD; Sherwin, John E. PhD
From Comprehensive Genetics, Fetal Medicine Foundation of America, and the Department of Obstetrics and Gynecology, Mount Sinai School of Medicine, New York, New York; and the Perkin Elmer/NTD Laboratories, New York, New York.
See related editorial on page 806.
Presented at the Annual Meeting of the Society for Maternal Fetal Medicine, February 4–6, 2010, Chicago, Illinois.
Corresponding author: Mark I. Evans, MD, Comprehensive Genetics, 131 E. 65th Street, New York, NY 10065; e-mail: Evans@compregen.com.
Financial Disclosure Dr. Evans is a consultant for Perkin Elmer/NTD and is a member of the nuchal translucency oversight committee for the Society for Maternal–Fetal Medicine and the Fetal Medicine Foundation. Drs. Sherwin, Hallahan, and Krantz are full-time employees of Perkin Elmer/NTD Laboratories, which performs the biochemical testing and coordinates the nuchal translucency measurement incorporation into algorithms.
OBJECTIVE: To analyze the maximum nuchal translucency from 327 centers to determine whether a more-than-expected number of centers had maximum nuchal translucency of 2.5 mm or less (approximately 4% of nuchal translucency values should be 2.5 mm or higher).
METHODS: We analyzed data from 182,669 nuchal translucency cases at centers in which at least 100 nuchal translucency examinations were performed from July 2008 through June 2009 and investigated the appropriateness of the distribution of values. We then investigated the likelihood of the skewing of the distribution seen using a 100 simulations of such modeled data.
RESULTS: Based on a binomial distribution, the chance that a center would have no nuchal translucency values above 2.5 mm is 1.7% for 100 patients per center, and 0.2% for 150 patients per center. Additionally, the median multiples of the median should shift by approximately 2.5% if all nuchal translucency values higher than 2.5 mm are excluded from the population. Our data show that 7.3% of centers had a maximum nuchal translucency of to 2.5 mm or less, and more than 20% have never reported an nuchal translucency of greater than 3 mm. The maximum nuchal translucency at a center correlated positively with its median multiple of the median. Centers with no nuchal translucency values greater than 2.5 mm also have nearly 50% of their ultrasonographers with excessive low nuchal translucency (greater than 10% of cases less than fifth percentile).
CONCLUSION: Too many centers have a maximum nuchal translucency of 2.5 mm or lower, low median nuchal translucency, and excessive low nuchal translucency, indicating that data from these centers are not representative of the expected distribution of nuchal translucencies. Our data suggest a systematic undermeasurement of nuchal translucency.
LEVEL OF EVIDENCE: III
Nuchal translucency quality review in the United States has been usually conducted by obtaining nuchal translucency measurements from laboratories that perform the biochemistry.1 For this quality review to be accurate, however, it is important to ensure that the nuchal translucency data provided are representative of a general screening population.2
Considerable data have shown that inexperienced ultrasonographers tend to undermeasure nuchal translucencies, and that even an undermeasurement of 0.5 mm can reduce the sensitivity of screening programs by about 18%.3 The effect of undermeasurement is significant throughout the range of nuchal translucency values; however, it is most critical for higher values that impart the greatest risk. Such measurements when rigorously monitored with continual feedback in clinical trials such as the Biochemistry and Fetal Nuchal Translucency Screening (BUN) trial and the First and Second Trimester Evaluation of Risk (FASTER) trial tend to improve over time.4,5 However, such documentation is lacking in actual practice in areas where there has been less-than-optimal continuing feedback to providers.
In the United States in actual practice, quality review programs have not been as proactive in feedback as in the United Kingdom and some of the European countries.3,6 Therefore, we decided to examine the distribution of nuchal translucency measurements by exploring the percentage of cases at the extremes of the curve. We analyzed the maximum nuchal translucency by center to determine whether there was a higher than expected number of centers with maximum nuchal translucency of 2.5 mm or less (approximately 4% of nuchal translucency values should be above 2.5 mm).
MATERIALS AND METHODS
We analyzed 182,669 nuchal translucency examinations from 327 centers in which at least 100 examinations were performed from July 2008 to June 2009. Cases submitted to our laboratory by other laboratories to run the assays that maintained their own relationships with quality review organizations were excluded. The average gestational age at ultrasound examination was 87.7 days (standard deviation [SD]=4.16). The average maternal age was 31.84 years (SD=5.60). The population was 57.3% white, 13.0% Hispanic, 11.8% African American, 7.7% Asian, and 10.3% other ethnic groups. Among the 327 centers in the study, 41% performed fewer than 250 examinations, 26% performed between 250 and 499 examinations, 19% performed between 500 and 999 examinations, and 13% performed more than 1,000 examinations.
For each center, the maximum nuchal translucency, median sample size (n), and median nuchal translucency multiples of the median (MoM) by center were also determined. Data were then grouped into categories based on the maximum nuchal translucency. Excessive low nuchal translucencies were defined as more than 10% of cases measuring lower than the fifth percentile. Nuchal translucency percentiles were determined based on the mixed model distribution described by Wright et al.7
The observed results for the percentage of centers with maximum nuchal translucency of 2.5 mm or lower were compared with expected results that would be expected in a population in which nuchal translucency measurement followed the general population. The comparison was performed using the binomial distribution with P=.04 for a fixed number of examinations and by using simulation where the number of examinations was equal to the number of examinations at each of the 327 centers. In the simulation, a random number between 0 and 1 was generated. If the random number was less than 0.04, the simulated examination was considered to have a nuchal translucency greater than 2.5 mm. Data were then grouped by center, and it was determined how many centers had no nuchal translucency values above 2.5 mm. The simulation was repeated 100 times.
Our data show that more than 20% of centers have never reported a nuchal translucency of more than 3 mm, and 7.3% of centers had maximum nuchal translucencies of 2.5 mm or less. (Table 1). Based on a binomial distribution, the chance that a center would have no nuchal translucency value above 2.5 mm is 1.7% for 100 patients per center and 0.2% for 150 patients per center. A simulation trial showed that out of 100 trials, 57% of the time there would be no centers with a maximum nuchal translucency below 2.5 mm, and in only one trial were there as many as four of the 327 centers with a maximum nuchal translucency less than 2.5 mm in any one trial (Fig. 1).
The maximum nuchal translucency at a center correlates positively with its median MoM (Spearman rank correlation=.1574, P=.004). If all nuchal translucency values higher than 2.5 mm were excluded from the population because of presumed immediate referral to tertiary centers with no bloods being obtained, the median MoM should shift by only approximately 2.5%. Centers with no nuchal translucency values more than 2.5 mm also have nearly 50% of their ultrasonographers with excessive low nuchal translucencies (Table 1, Fig. 2).
There was a small positive, but not significant, correlation between the number of examinations performed and the median nuchal translucency MoM observed. The Spearman rank correlation coefficient between median nuchal translucency MoM and number of examinations performed was 0.067 (P=.226). However, there was a very highly significant correlation between the median N and the maximum nuchal translucency (Spearman rank correlation=0.4385, P<.001). We then looked at the distribution of the extremes, which confirmed that those centers with low mean nuchal translucency have both an excess of very low and diminished very high nuchal translucencies, thus suggesting a pattern of performance rather than random events (Table 2).
Evaluation of the maximum nuchal translucency may be a key indicator in assessing performance of nuchal translucency. In some cases a low maximum nuchal translucency may indicate that the center is using a cutoff over which biochemistry is not performed. Mechanisms should be in place to capture these data so that evaluation of quality control can be optimized. Preferably, performing biochemistry before nuchal translucency would resolve this issue and also enhances screening performance.1,2 Table 2 shows that centers with very low maximum nuchal translucencies have a vast overrepresentation of cases less than the fifth percentile, suggesting poor technical performance of measurements rather than a failure to perform biochemistry. Statistically, too many centers have a maximum nuchal translucency lower than 2.5 mm, low median nuchal translucency, and excessive low nuchal translucencies, indicating that the data from these centers are not representative of the expected general screening distribution of nuchal translucencies. Such data are very problematic for the optimal performance of screening in the United States.
Some centers may not be obtaining bloods from patients with high nuchal translucencies or referring such patients directly to tertiary centers. However, our data suggest a systematic undermeasurement of nuchal translucencies. The centers tend to be ones with lower volumes, raising the question of minimum cases per year to provide adequate performance.
Overall, our data suggest a serious problem in the diffusion of nuchal translucency utilization and effectiveness throughout the United States that should be addressed by methods to improve the performance of nuchal translucency measurements—particularly at centers with lower volumes. Such has been a common problem of the introduction of new technologies8,9 It is well appreciated in the laboratory industry for decades that quality control of all analytes is critical to adequate performance.10 If nuchal translucency measurements are to be used in laboratory algorithms to determine clinical risks, then they need to conform to the same standards as those laboratory measurements. Poor coefficients of variation from deviation of nuchal translucency measurements will lead to underperformance of screening.
Quality assurance of nuchal translucency measurements has focused on the individual ultrasonographer, who passes a written examination and submits a number of nuchal translucency examination images for review. Although this is an important component of quality assurance, it does not go far enough. Anyone involved in this process knows that the ultrasonographer “cherry picks” which images will be submitted for review to the certification organization. This merely certifies that the ultrasonographer possesses the requisite knowledge and skill to properly perform the nuchal translucency measurement. This is analogous to first obtaining a driver's license. It doesn't mean that the operator is immediately a “good driver.” Analysis of real patient data assesses performance under “field conditions.” Although analysis of individual ultrasonographer data is essential and currently is conducted by the nuchal translucency quality review organizations, evaluation of center data with a higher number of examinations can reveal additional trends that would not be apparent by looking at smaller numbers on an ultrasonographer-by-ultrasonographer basis.
Center specific medians have been proposed to “normalize” skewed distributions, but such would only mask the identification of poor performance.11 We would discourage this practice, because without the opportunity to recognize a problem, it cannot be addressed. Current ultrasonographers at centers with low median MoMs and high percentages of cases under the 5th percentile should undergo some form of retraining, and those ultrasonographers who are unable or unwilling to improve performance should no longer be allowed to assess nuchal translucency.
Alternatively, as part of a remedy, in other work one of us (M.I.E.) has proposed a mathematical adjustment to the likelihood ratio calculation that would incorporate the ultrasonographer's performance (abstract published). Thus, each ultrasonographer would have a “handicap” score that will partially address the issue of high variability among performers. However, key to any such method is the continual feedback as originally suggested in the industry by Demming to improve performance.12 Without such, all efforts to improve screening methodology will come up short.
1. Evans MI, Krantz DA, Hallahan TW, Galen RS. Meta-analysis of first trimester Down Syndrome screening studies: free beta-human chorionic gonadotropin significantly outperforms intact human chorionic gonadotropin in a multimarker protocol. Am J Obstet Gynecol 2007;196:198–205.
2. Evans MI, Cuckle HS. Biochemical screening for aneuploidy. Expert Rev Obstet Gynecol 2007;2:765–73.
3. Evans MI, Van Decruyes H, Nicolaides KH. Nuchal translucency measurements for first-trimester screening: the ‘price’ of inaccuracy. Fetal Diagn Ther 2007;22:401–4.
4. Wapner R, Thom E, Simpson JL, Pergament E, Silver R, Filkins K, et al; First Trimester Maternal Serum Biochemistry and Fetal Nuchal Translucency Screening (BUN) Study Group. First-trimester screening for trisomies 21 and 18. N Engl J Med 2003;349:1405–13.
5. Malone FD, Canick JA, Ball RH, Nyberg DA, Comstock CH, Burkowski R, et al. First-trimester or second-trimester screening, or both, for Down's syndrome. N Engl J Med 2005;353:2001–11.
6. Evans MI, Pergament E. Impact of quality of nuchal translucency measurements on detection rates of trisomies 13 and 18. Fetal Diagn Ther 2010;27:68–71.
7. Wright D, Kagan KO, Molina FS, Gazzoni A, Nicolaides KH. A mixture model of nuchal translucency thickness in screening for chromosomal defects. Ultrasound Obstet Gynecol 2008;31:376–83.
8. Evans MI, Hanft RS. The introduction of new technologies. ACOG Clin Sem 1997;2:1–3.
9. Cohen AB, Hanft RS. Technology in American health care: policy directions for effective evaluation and management. Ann Arbor (MI): University of Michigan Press; 2004.
10. Galen RS, Gambino SR. Beyond normality: the predictive value and efficiency of medical diagnosis. New York (NY): Wiley; 1975.
11. Palomaki GE, Lee JE, Canick JA, McDowell GA, Donnenfeld AE; ACMG Laboratory Quality Assurance Committee. Technical standards and guidelines: prenatal screening for Down syndrome that includes first-trimester biochemistry and/or ultrasound measurements. Genet Med 2009;11:669–81.
12. Demming WE. Out of the crisis. Cambridge (MA): MIT Press; 1986.
© 2010 by The American College of Obstetricians and Gynecologists.