Skip Navigation LinksHome > January 2005 - Volume 16 - Issue 1 > Optimal Cut-point and Its Corresponding Youden Index to Disc...
Epidemiology:
doi: 10.1097/01.ede.0000147512.81966.ba
Original Article

Optimal Cut-point and Its Corresponding Youden Index to Discriminate Individuals Using Pooled Blood Samples

Schisterman, Enrique F.; Perkins, Neil J.; Liu, Aiyi; Bondell, Howard

Free Access
Supplemental Author Material
Article Outline
Collapse Box

Author Information

From the Division of Epidemiology, Statistics and Prevention Research, National Institute of Child Health and Human Development, National Institutes of Health, DHHS, Bethesda, Maryland; >Department of Mathematics and Statistics, American University, Washington, DC; ‡Department of Statistics, Rutgers University, Piscataway, New Jersey.

Supplemental material for this article is available with the online version of the Journal at www.epidem.com.

Submitted 12 September 2004; final version accepted September 21, 2004.

Supported by Intramural resources from NICHD.

Correspondence: Enrique F. Schisterman, Division of Epidemiology, Statistics and Prevention Research, National Institute of Child Health and Human Development (NICHD), 6100 Executive Blvd., Bethesda, MD 20852. E-mail: schistee@mail.nih.gov.

Collapse Box

Abstract

Costs can hamper the evaluation of the effectiveness of new biomarkers. Analysis of smaller numbers of pooled specimens has been shown to be a useful cost-cutting technique. The Youden index (J), a function of sensitivity (q) and specificity (p), is a commonly used measure of overall diagnostic effectiveness. More importantly, J is the maximum vertical distance or difference between the ROC curve and the diagonal or chance line; it occurs at the cut-point that optimizes the biomarker's differentiating ability when equal weight is given to sensitivity and specificity. Using the additive property of the gamma and normal distributions, we present a method to estimate the Youden index and the optimal cut-point, and extend its applications to pooled samples. We study the effect of pooling when only a fixed number of individuals are available for testing, and pooling is carried out to save on the number of assays. We measure loss of information by the change in root mean squared error of the estimates of the optimal cut-point and the Youden index, and we study the extent of this loss via a simulation study. In conclusion, pooling can result in a substantial cost reduction while preserving the effectiveness of estimators, especially when the pool size is not very large.

Back to Top | Article Outline

ArticlePlus

Click on the links below to access all the ArticlePlus for this article.

Please note that ArticlePlus files may launch a viewer application outside of your web browser.

* http://links.lww.com/EDE/A115

* http://links.lww.com/EDE/A116

* http://links.lww.com/EDE/A117

* http://links.lww.com/EDE/A118

The current emphasis on early detection and prevention of chronic and acute diseases has led to development of new and sophisticated biomarkers. However, costs of evaluation of their effectiveness can sometimes limit the feasibility of such testing. For example, the interleukin-6 biomarker of inflammation has been suggested to have potential discriminatory ability for myocardial infarction. However, the cost of a single assay can be so high (20 or more times greater than storage and technician costs) that financial considerations will hinder attempts to evaluate the usefulness of the biomarker. Analysis of results based on smaller numbers of pooled specimens has been shown to be a useful cost-cutting technique, especially with microarray experiments.1–5

By pooling (ie, physically combining individual specimens), the amount of information per assay is increased while the number of assays needed to evaluate this information decreases.4,6 We assumed that measurement of the samples being pooled adequately represents the average of the individual unpooled sample. Conveniently, this is often the case, as most tests are expressed per unit of volume.

Receiver operating characteristics (ROC) curves are used in biomedical research to evaluate the effectiveness of biomarkers for distinguishing individuals with disease from those without.7–10 The Youden index (J), a function of sensitivity (q) and specificity (p), is a commonly used measure of overall diagnostic effectiveness.7–13 This index ranges between 0 and 1, with values close to 1 indicating that the biomarker's effectiveness is relatively large and values close to 0 indicating limited effectiveness. Figure 1 shows that J is the maximum vertical distance or difference between the ROC curve and the diagonal or chance line. J is defined by

Figure 1
Figure 1
Image Tools

over all cut-points c, −∞ < c < ∞. If risk of disease is a monotonically increasing function of the marker level, sensitivity decreases and specificity increases with rising c. Thus, there is a penalty, decreased specificity for increasing sensitivity too far. J occurs at the optimal cut-point for calling a patient diseased, maximizing the number of correctly classified individuals.11–14 On the other hand, the consequences of a positive or negative test result (ie, intervention) may be quite different and the loss from missing a case may be greater than from overcalling a control. Then, a differential weighting is needed to optimize J (Appendix A.1, available with the electronic version of the article).

With equal weight given to errors of sensitivity and specificity, J can be determined graphically by plotting fx and fy, the probability density functions of the cases and controls, respectively, for a continuously distributed biomarker (Fig. 2). J is the difference of the area under fx and fy to the right of the cut-point, with negative area when fy > fx. This area is identical to the difference of the area under fy and fx to the left of the cut-point. With unequal weights, in a ratio R, J can be seen as the difference between fy and Rfx.

Figure 2
Figure 2
Image Tools

For diagnostic purposes and decision-making, health practitioners dichotomize continuous biomarkers into healthy and diseased patients. The optimal cut-point, the value used to separate these groups, and J occur at an intersection between the probability density functions of cases and controls such that

for some small ε > 0. This is true when the mean of the cases is greater than that of the controls. The second criterion is necessary when multiple intersections exist (for the proof, see Appendix A.1). From now on, c will denote the optimal cut-point resulting in J.

Equation (Uncited)
Equation (Uncited)
Image Tools

Due to the high costs entailed by some biomarkers, several authors have proposed the use of pooling, and have evaluated ROC curve analysis when dealing with such data.6,15 However, the effect of pooling on J and c has yet to be explored.

In this paper, we extend the work of Faraggi et al6 and Liu and Schisterman15 to evaluation of c and J under various distributional assumptions. We examine the effect of pooling on the efficiency of c and J estimation. In the circumstances when a fixed number of individuals are available for testing and samples are pooled to reduce the number of assays (thus lowering the costs), a loss of information is expected. We measure this loss of information by the change in root mean squared error (RMSE) of the estimate of c and J, and examine the extent of this loss via a simulation study.

We also examine the situation where the number of assays available is fixed and pooling is used to increase the information per assay. This procedure may improve the accuracy of the estimate and is specifically applicable in cases where assaying cost significantly exceeds the cost of obtaining samples.

Back to Top | Article Outline
Inference on Youden Index and Optimal Cut-point Based on Pooled Data: Normal Assumptions

Assume that the responses of a specific biomarker are normally distributed, such that the cases (X), or true positives, have mean μx and variance σx2, and the controls (Y), or true negatives, have mean μy and variance σy2, and μx > μy. For μx < μy, one may simply switch the cases with controls in the following analysis. Under these assumptions, sensitivity (q(c)) and specificity (p(c)) can be written as

for a given cut-point c, where Φ denotes the standard normal distribution function. Accordingly, test measurements falling below c are negative results and those at or above c are positive.

As stated in the previous section, the optimal cut-point (and thus J) occurs at an intersection of the probability density functions of cases and controls. The number of intersections is a function of the variances of the cases and controls. One simple case is that of equal variance in cases and controls, σx2 = σy2, where only one intersection exists and c is simply the midpoint between the means, (μy − μx)/2. In the case of unequal variance, the intersections can be found by the following quadratic equation:

where a = μx − μy and b = σxy. To find J, let us first order the intersections, c1 < c2. If b > 1, then J occurs at c2; alternatively, if b < 1, then J occurs at c1. Now, using Eq 3 for sensitivity and Eq 4 for specificity, J can be found from the appropriate cut-point.

Equation 3
Equation 3
Image Tools
Equation 4
Equation 4
Image Tools

When data on both cases and controls are available, appropriate estimates for μx, σx2, μy, σy2 can be calculated. These parameter estimates, when substituted into Eq 3, Eq 4, and Eq 5, yield the estimate ĉ and subsequently Ĵ.

Equation 5
Equation 5
Image Tools

Suppose that the data available are in the form of pooled samples, obtained as follows. First, individuals of similar disease status (ie, cases with cases, controls with controls) are randomly placed into groups of size g. Then, grouped individual specimens are combined as pooled samples and are tested as single observations. Assuming that the specimens are measured per unit of volume, a pool's measurement is then considered as the average of the member's measurements. This has been shown to be a reasonable assumption.4

Consider the instance where there are n and m pooled observations available of cases and controls, respectively, with groups of size g. Let XPi, i = 1,...,n, denote cases, and YPj, j = 1,...,m denote controls such that

Consequently, from the additive property of the normal distribution, we have μxp = μx, σ2xp = σ2x/g, μyp = μy, σ2yp = σ2y/g.

Equation (Uncited)
Equation (Uncited)
Image Tools
Equation (Uncited)
Equation (Uncited)
Image Tools

Using the usual notation, let (xp), Sxp, (yp) and Syp denote the standard estimates of μxp, σxp, μyp, and σyp, respectively. The parameters of the unpooled distributions can then be estimated accordingly by μx = (xp), σx = √g Sxp, μy = (yp) and σy = √gSyp. Substituting these estimates for the parameters in the above equations yields the estimates ĉ and Ĵ based on pooled data.

Back to Top | Article Outline
Inference on Youden Index and Optimal Cut-point Based on Pooled Data: Gamma Assumptions

The assumption of normality is often not justifiable in practice. Some biomarkers are skewed right and are best represented by some form of the gamma distribution. Suppose that the responses to a given biomarker follow a gamma distribution such that cases are gamma(αX, βX) and controls are gamma(αY, βY). Based on these distributional assumptions, sensitivity (q(c)) and specificity (p(c)) can be calculated as shown in Appendix A.2 (available with the electronic version of the article).

As with the normal case, J is realized at an intersection of the probability density functions. When case and control responses follow a gamma distribution, a single intersection frequently exists, the location of which defines the optimal cut-point c. Some special cases are

1. αx = αy = α, then

2. βx = βy = β, then

Equation (Uncited)
Equation (Uncited)
Image Tools

Otherwise, the intersection must be found numerically. When 2 intersections exist, c is located by the previous criteria (Eq 2). Now, J can be calculated at c by substituting Eq 6 and Eq 7 (Appendix A.2) for sensitivity and specificity in Eq 1.

Equation 1
Equation 1
Image Tools
Equation 2
Equation 2
Image Tools
Equation (Uncited)
Equation (Uncited)
Image Tools

Suppose again that pooled specimens are available with the pooling process being the same as defined earlier. Again, we let XPi, i = 1,...,n and YPj, j = 1,...,m denote the pooled observations of cases and controls, respectively. Using the additive property of the gamma distribution, we have

and

Equation (Uncited)
Equation (Uncited)
Image Tools

each having a pooling size g. We will continue to assume that the measure of a pooled observation is the mean of the g unpooled measures. Consequently, αXP = g× αX, αYP = g× αY, βXP = βX/g and βYP = βY/g.

Equation (Uncited)
Equation (Uncited)
Image Tools

The maximum likelihood estimates (α̂)PX, (β̂)PX, (α̂)PY and (β̂)PY can be obtained numerically from the observations on the pooled specimens. Using these estimators and the association between the distributions of pooled and unpooled observations, estimates of the unpooled distribution parameters can be obtained by (α̂)X = (α̂)PX/g, (β̂)X = g× (β̂)PX, (α̂)Y = (α̂)PY/g, and (β̂)Y = g× (β̂)PY. The estimates ĉ and Ĵ can now be obtained by substituting these estimates for the parameters and following the steps outlined previously.

Back to Top | Article Outline
Simulation Study

To fully explore the effects of pooling on the estimates ĉ and Ĵ, we conducted simulation studies by generating data (cases and controls) from either normal or gamma distributions. Since ĉ and Ĵ are a function of the intersection of the probability density functions for cases and controls, the parameters selected represent a wide variety of distributional conditions (normal and gamma) exemplified by different levels of separation (J = 0.2, 0.4, 0.6, 0.8). While simulations at all J levels are presented, analysis will focus primarily on J of 0.6 and 0.8, or the “useful”, better diagnostic biomarker levels. Our simulations were limited to pooled size of 2 or 4 because pooling sizes of 5 and above result in a loss of identifiable skewness, due to the central limit theorem. A summary of our investigation is presented in Tables 1-4 We considered 2 common general conditions regarding availability of samples in an experimental setting.6 The first involves fixing the number of study subjects (N=M= 40,100,200), and the second fixes the number of assays (n = m = 40,100,200). We generated 2000 individual samples from each set of parameters. Percent bias and relative root mean squared error (RMSE) were then determined by comparing estimates to the true c and J (calculated using the true parameter values) as follows:

Table 1
Table 1
Image Tools
Table 2
Table 2
Image Tools
Table 3
Table 3
Image Tools
Table 4
Table 4
Image Tools

where ĉ is the estimated optimal cut-point. %Bias(Ĵ) was calculated in the same manner; and

Equation (Uncited)
Equation (Uncited)
Image Tools

where the

Equation (Uncited)
Equation (Uncited)
Image Tools

is the

Equation (Uncited)
Equation (Uncited)
Image Tools

for unpooled data, and

Equation (Uncited)
Equation (Uncited)
Image Tools

is

Equation (Uncited)
Equation (Uncited)
Image Tools

for pooled data of size 2 or 4.

Equation (Uncited)
Equation (Uncited)
Image Tools

The first condition, when the number of subjects available is fixed, looks at the degradation of the estimate ĉ as pooling size increases (g = 1,2,4) resulting in a decrease in the number of tested samples—n = N/g. For instance, 40 control unpooled specimens are converted to 20-pooled specimens with each specimen consisting of a randomly chosen pair of controls (g = 2) or are converted to 10-pooled specimens with each specimen consisting of randomly chosen tetrad of controls (g = 4). The same procedure is applied to the case population.

Under normality assumptions (Table 1), the percent bias in the estimate of the optimal cut-point was negligible on all levels of discrimination and pooling, even for small sample sizes. As expected, the relative RMSE was inversely associated with the pooled size. No considerable distinction could be made between the RMSE from un-pooled data (g = 1) and pooled data (g = 2), J = 0.6, 0.8. However, for g = 4, the relative loss of efficiency is 3 times that of pairs. This is the effect of central limit theorem and is to be expected when cutting the sample by 75%.

In the gamma case (Table 2), the percent bias and relative RMSE increase in magnitude as g increases and, consequently, n and m decreases. The increase in bias due to pooling is negligible for all J = 0.4, 0.6, and 0.8. Relative RMSE increase for g = 2 are on par with that of the normal case, but g = 4 results are consistently 10% higher than the normal tetrads. The positive bias for both estimates based on the unpooled as well as on the pooled data greatly attenuates as sample size is increased. This is a result of using maximum likelihood estimators to estimate the optimal cut-point under small samples. Moreover, the bias is largely reduced for J = 0.4, 0.6, and 0.8 even for small sample size, which are actually the markers of scientific interest.

Biomarkers with poor distinguishing ability (eg, J = 0.2), also behave poorly under pooling. For example, when 40 unpooled samples are pooled in pairs, the RMSE increases by 37% for the gamma case. More generally, this relationship is true for both normal and gamma cases.

Under the second condition, when the number of assays to be performed is fixed, pooling effectively increases the overall sample size and the amount of information, via an increase in N(N = n · g) (Tables 3 and 4).

Again, bias remains unaffected, less than 1% bias for all levels of pooling for “useful” J. As pooling size increases, there is a consistent reduction in RMSE. For the normal case, as the level of pooling increases (g = 1,2,4), the RMSE for “useful” J substantially decreases (about half for pools of 4). Likewise, under gamma assumptions, as the level of pooling increases, g = 1,2,4, the benefits in RMSE are substantial (40% decrease for pools of 4). Pooling when J = 0.2 and 0.4 reveals a less dramatic benefit in RMSE.

These methods provide a useful tool for making inferences about unpooled samples when assays are based on pooled specimens. This is more clearly seen through use of an example, as illustrated below.

Back to Top | Article Outline
Example

Evidence shows that inflammation may play a contributing role in the development of coronary heart disease (CHD). Interleukin-6 has been linked with the presence of infections in the vessel wall and with atherosclerosis.16,17 Moreover, epidemiologic data show that infection in remote sites in the etiology of CHD.

Individual measurements of interleukin-6 on 80 volunteers were obtained at Cedars-Sinai Medical Center. Forty individuals who recently (within 2 weeks from the event) survived a myocardial infarction (MI) were defined as cases, after being confirmed by rest electrocardiogram (ECG) and laboratory measurements; the remaining 40 subjects served as controls. The controls had a normal rest ECG, were free of symptoms and had no previous cardiovascular procedures or MIs. In addition, the blood specimens were randomly pooled in groups of 2 and 4, for the cases and the controls separately, and remeasured. Faraggi et al6 have shown, using the same data, that for interleukin-6 the assumption that the pooled sample measurements are the equivalent of the average of the individual cases is justified. Due to the costs involved such confirmatory evidence for the averaging assumption will generally not be available.

Distributional assumptions were also tested and found to fit well with gamma assumptions, confirming the findings of Faraggi and coauthors.6 The mean (± SD) in the control and case unpooled samples, respectively, were 1.85 (±1.37) and 4.29 (±2.18). Youden index and cut-point were estimated using the method described previously under gamma assumptions. Table 5 shows that the Youden index was approximately 0.5 for unpooled and pooled data. More importantly, the optimal cut-point was estimated to be 2.41 for unpooled data and was not very much affected by pooling, as shown in Figure 1 and 3. A 95% bootstrapped confidence interval based on unpooled data was estimated to be 1.8 to 3.6, containing both estimates (2.06 [g = 2] and 2.70 [g = 4]) based on pooled data, despite the small number of specimens.

Table 5
Table 5
Image Tools
Figure 3
Figure 3
Image Tools
Back to Top | Article Outline

Discussion

In this paper, we have presented a method to estimate the Youden index and the optimal cut-point and extended its applications to pooled samples. We extend the work of Faraggi et al6 and Liu and Schisterman15 to the cut-point, c, and Youden Index, J, under various distributional assumptions. We have shown that pooling is a statistically viable cost-saving approach, through a reduction in the number of assays required, especially with pool sizes of 2 and 4.

Most other statistical methods currently available for the analysis of biomarkers deal with comparison of proportions between cases and controls and power analysis, eg, for a genotype.4,19 Our methods are specific for continuous data, where finding the optimal cut-point an important issue.

Back to Top | Article Outline
Relation Between Youden Index and The Likelihood Ratio

It is of interest to note that, since the Youden index of a continuous biomarker is a function of sensitivity and specificity, its relation to the likelihood ratio positive and negative may be useful. Graphically, the likelihood ratio positive (LR+) is the slope (q/(1− p)) of the line through the origin and a point on the ROC curve, while the likelihood ratio negative (LR−) is the slope ((1− q)/p) of the line through (1,1) and the same point on the ROC curve. The product of the likelihood ratios [q(1− q)/p(1 − p)] is the slope of the angle bisector. The Youden index, J, is the point at which the product of the two-likelihood ratio is equal to 1 or when the tangent to the ROC curve is parallel to the chance line (Fig. 1). Also, confidence intervals for c and J can be easily obtained using bootstrap methods and statistical software that is currently available.18

Back to Top | Article Outline
Pooling Assumptions

Correct implementation of the method developed in this paper requires assumptions, if the researcher sees only the pooled data. The first assumption is that the value obtained from a pooled assay can be considered to be the average of the individual values of the pooled specimens. There is both a biologic and a methodological aspect to this assumption. Biologically, this assumption can be deemed reasonable based on expert knowledge of the biomarker. If, for example, because of the molecular structure of the biomarker, pooling blood samples might yield a statistic other than the average (eg, maximum), then this methodology is inappropriate for the evaluation of the optimal cut-point and the Youden index. On the other hand, when this assumption is reasonable biologically, differences between the pooled sample and the average of individual specimens is due to “random measurement error,” defined as the random variability that led to inaccuracy in the estimation of the true mean value. For instance, if the volume of the individual specimens to be pooled is not equal, the pooled sample will result in a weighted average of the volume per value of the biomarker. Therefore, for normally distributed biomarkers, the addition of mean zero measurement error and variance σ&epsiv;2 will affect the estimates of ĉ one of 3 different ways depending on the ratio between σX2Y2. If σX2Y2 = 1, then ĉ will remain unbiased, because the location where the 2 distributions intercept would remain unchanged (Fig. 2). If σX2Y2 > 1, then ĉ will be positively biased and similarly if σX2Y2 < 1 then ĉ will be negatively biased. For biomarkers that follow a gamma distribution, measurement error will always cause a positive bias in ĉ. This is due to the dependent relationship between the mean and variance of gamma distributions. Also, measurement error always results in an attenuation of Ĵ. Since J is a measure of differentiation between cases and controls, it is intuitive that when error is introduced the ability to differentiate decreases.

Back to Top | Article Outline
Distributional Assumptions

The second assumption is that the unpooled biomarkers follow a known parametric distribution. A more formal evaluation of distributional assumption would be possible using a moment-based estimating-equation approach to deal with situations where likelihood functions based on pooled data are difficult to work with. We outlined the method to obtain estimates and test statistics of the parameters of interest in the general setting. We demonstrated the approach on the family of distributions generated by the Box-Cox transformation model, and, in the process, construct tests for goodness of fit based on the pooled data. Nevertheless, in our experience, the researcher will often develop some sense of both these assumptions during the early stages of the biomarker development by means of a validation study.

Back to Top | Article Outline
Pooling Size

Pooling sizes of 5 and above, while fiscally attractive, are prone to 2 difficulties. The first is a consequence of the central limit theorem; averages tend to be more normally distributed as sample size increases. Identifying a biomarker's un-pooled distribution is difficult because the central limit theorem hinders our ability to distinguish between a skewed and a symmetric distribution. The second difficulty arises only when a fixed number of subjects are reduced to an unreasonably small sample size due to pooling and rendering the parameter estimation unreliable. For instance, in the example presented above, we had 40 cases and 40 controls contributing blood samples. If g = 10, then we are left with 8 assays (4 cases and 4 controls) on which to estimate the means and standard deviations necessary for ĉ and Ĵ.

Back to Top | Article Outline
Implications

This method is relevant to studies of markers for early detection and prevention of disease and for studies of markers of exposure and disease in molecular epidemiology when, for example, deciding whether a biomarker is worth pursuing further or is ready for a study. Furthermore, once this method is applied and a biomarker demonstrates discriminatory ability, the optimal cut-point can be used in clinical practice to classify patients as healthy or diseased, after proper validation.

In summary, we showed that estimating c and J under pooling is a cost-effective, statistically sound approach for evaluating biomarkers. Such estimation has potential applications for research and clinical practice and for hypothesis development.

Back to Top | Article Outline

REFERENCES

1.Farrington C. Estimating prevalence by group testing using generalized linear models. Stat Med. 1992;11:1591–1597.

2.Tu X, Litvak E, Pagano M. On the informativeness and accuracy of pooled testing in estimating prevalence of a rare disease: application to HIV screening. Biometrika. 1995;82:287–297.

3.Barcellos L, Klitz W, Field L, et al. Association mapping of disease loci, by use of a pooled DNA genomic screen. Am J Hum Genet. 1995;61:734–747.

4.Weinberg CR, Umbach DM. Using pooled exposure assessment to improve efficiency in case-control studies. Biometrics. 1999;55:718–726.

5.Kemdziorski CM, Zhang Y, Lan H, Attie AD. The efficiency of pooling mRNA in micro array experiments. Biostatistics. 2003;4:465–477.

6.Faraggi D, Reiser B, Schisterman EF. ROC curve analysis for biomarkers based on pooled assessments. Stat Med. 2003;22:2515–2527.

7.Zou KH, Hall WJ, Shapiro DE. Smooth non-parametric receiver operating characteristic (ROC) curves for continuous diagnostic tests. Stat Med. 1997;16:2143–2156.

8.Zweig MH, Campbell G. Receiver operator characteristic (ROC) plots; a fundamental evaluation tool in clinical medicine. Clin Chem. 1993;39:561–577.

9.Goddard MJ, Hinbery I. Receiver operator characteristic (ROC) curves and non-normal data: an empirical study. Stat Med. 1990;9:325–337.

10.Wieand S, Gail MH, James BR, James KL. A family of non-parametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika. 1989;76:585–592.

11.Youden WJ. An index for rating diagnostic tests. Cancer. 1950;3:32–35.

12.Barkan N. Statistical inference on r*specificity + sensitivity. Doctoral Dissertation 2001. Haifa University.

13.Bamber DC. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psychol. 1975;12:387–415.

14.Hilden J, Glasziou P. Regret graphs, diagnostic uncertainty and Youden's Index. Stat Med. 1996;15:969–986.

15.Liu A, Schisterman EF. Comparison of diagnostic accuracy of biomarkers with pooled assessments. Biom J. 2003;45:631–644.

16.Chilton RJ. Recent discoveries in assessment of coronary heart disease: impact of vascular mechanisms on development of atherosclerosis. J Am Osteopath Assoc. 2001;101:S1–S5.

17.Yudkin JS, Kumari M, Humphries SE, Mohamed-Ali V. Inflammation, obesity, stress and coronary heart disease: is interleukin-6 the link? Atherosclerosis. 2000;148:209–214.

18.Carpenter J, Bithell J. Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Stat Med. 2000;19:1141–1164.

19.Peng X, Wood CL, Blalock EM, et al. Statistical implications of pooling RNA samples for microarray experiments. BMC Bioinformatics. 2003;24:4–26.

Back to Top | Article Outline
Appendix 1

Assume that cases, X, and controls, Y, are represented by continuous unimodal distributions, and μy < μx. Let c0 be some cut-point and ci(I = 1,2) be the ith intersection of the probability density functions denoted by f. Youden index (J) is found by

The intervals for which fy > fx and fy < fx are determined by the variances of the distributions. Assuming σx2 > σy2 could result in 1 or 2 intersections. The 2-intersection case follows

Equation (Uncited)
Equation (Uncited)
Image Tools

For c0 in (−∞, c1)

Equation (Uncited)
Equation (Uncited)
Image Tools
Equation (Uncited)
Equation (Uncited)
Image Tools
Equation (Uncited)
Equation (Uncited)
Image Tools

Similarly, for c0 in (c1, c2)

Equation (Uncited)
Equation (Uncited)
Image Tools

And, for c0 in (c2, ∞)

Equation (Uncited)
Equation (Uncited)
Image Tools

Therefore

Equation (Uncited)
Equation (Uncited)
Image Tools

A similar argument proves that when a single intersection exists, the intersection is the cut-point for J. For the case where σx2 < σy2, this approach yields c1 as the optimal cut-point used for J.

Equation (Uncited)
Equation (Uncited)
Image Tools

Note: Using Figure 1 as a reference, it can be seen that moving the cut-point to the right would result in a loss in shaded area (Youden index). Since Youden index can be represented by the area between the 2 curves to either the right or left of the cut-point, moving the cut-point to the left also result in a decrease.

Cited By:

This article has been cited 58 time(s).

Veterinary Journal
Evaluation of semi-quantitative bone scintigraphy in canine elbows
Debruyn, K; Peremans, K; Vandermeulen, E; Van Ryssen, B; Saunders, JH
Veterinary Journal, 196(3): 424-430.
10.1016/j.tvjl.2012.10.022
CrossRef
Cancer Epidemiology Biomarkers & Prevention
Identification of PTHrP(12-48) as a Plasma Biomarker Associated with Breast Cancer Bone Metastasis
Washam, CL; Byrum, SD; Leitzel, K; Ali, SM; Tackett, AJ; Gaddy, D; Sundermann, SE; Lipton, A; Suva, LJ
Cancer Epidemiology Biomarkers & Prevention, 22(5): 972-983.
10.1158/1055-9965.EPI-12-1318-T
CrossRef
Ieee Transactions on Information Technology in Biomedicine
Fuzzy Logic-Based Prognostic Score for Outcome Prediction in Esophageal Cancer
Wang, CY; Lee, TF; Fang, CH; Chou, JH
Ieee Transactions on Information Technology in Biomedicine, 16(6): 1224-1230.
10.1109/TITB.2012.2211374
CrossRef
Biosensors & Bioelectronics
Electrochemical magneto immunosensor for the detection of anti-TG2 antibody in celiac disease
Kergaravat, SV; Beltramino, L; Garnero, N; Trotta, L; Wagener, M; Pividori, MI; Hernandez, SR
Biosensors & Bioelectronics, 48(): 203-209.
10.1016/j.bios.2013.04.012
CrossRef
Pharmaceutical Statistics
Confidence intervals for the difference in paired Youden indices
Zhou, HC; Qin, GS
Pharmaceutical Statistics, 12(1): 17-27.
10.1002/pst.1543
CrossRef
Preventive Medicine
Estimating the risk of peripheral artery disease using different population strategies
Grau, M; Baena-Diez, JM; Felix-Redondo, FJ; Fernandez-Berges, D; Comas-Cufi, M; Fores, R; Marrugat, J; Ramos, R
Preventive Medicine, 57(4): 328-333.
10.1016/j.ypmed.2013.06.007
CrossRef
Skin Research and Technology
Lesion classification using 3D skin surface tilt orientation
She, ZS; Excell, PS
Skin Research and Technology, 19(1): E305-E311.
10.1111/j.1600-0846.2012.00644.x
CrossRef
Biometrical Journal
Efficient design and analysis of biospecimens with measurements subject to detection limit
Vexler, A; Liu, AY; Schisterman, EE
Biometrical Journal, 48(5): 780-791.
10.1002/bimj.200610266
CrossRef
American Journal of Surgery
Severity of disease as main predictor for mortality in patients with pyogenic liver abscess
Chen, SC; Huang, CC; Tsai, SJ; Yen, CH; Lin, DB; Wang, PH; Chen, CC; Lee, MC
American Journal of Surgery, 198(2): 164-172.
10.1016/j.amjsurg.2008.08.022
CrossRef
Journal of Medical Screening
Is NESTROFT sufficient for mass screening for beta-thalassaemia trait?
Mamtani, M; Das, K; Jawahirani, A; Rughwani, V; Kulkarni, H
Journal of Medical Screening, 14(4): 169-173.

Journal of Computational Chemistry
In Silico Prediction and Screening of gamma-Secretase Inhibitors by Molecular Descriptors and Machine Learning Methods
Yang, XG; Lv, W; Chen, YZ; Xue, Y
Journal of Computational Chemistry, 31(6): 1249-1258.
10.1002/jcc.21411
CrossRef
Respiratory Care
Specific Conductance Criteria for a Positive Methacholine Challenge Test: Are the American Thoracic Society Guidelines Rather Generous?
Khalid, I; Morris, ZQ; DiGiovine, B
Respiratory Care, 54(9): 1168-1174.

Archives of Physical Medicine and Rehabilitation
Utility of Functional Status for Classifying Community Versus Institutional Discharges After Inpatient Rehabilitation for Stroke
Reistetter, TA; Graham, JE; Deutsch, A; Granger, CV; Markello, S; Ottenbacher, KJ
Archives of Physical Medicine and Rehabilitation, 91(3): 345-350.
10.1016/j.apmr.2009.11.010
CrossRef
Journal of Critical Care
Revised Acute Physiology and Chronic Health Evaluation score as a predictor of neurosurgery intensive care unit readmission: A case-controlled study
Lee, HF; Lin, SC; Lu, CL; Chen, CF; Yen, MF
Journal of Critical Care, 25(2): 294-299.
10.1016/j.jcrc.2009.12.007
CrossRef
Paediatric and Perinatal Epidemiology
To pool or not to pool, from whether to when: applications of pooling to biospecimens subject to a limit of detection
Schisterman, EF; Vexler, A
Paediatric and Perinatal Epidemiology, 22(5): 486-496.
10.1111/j.1365-3016.2008.00956.x
CrossRef
World Journal of Gastroenterology
Non-invasive diagnosis of gastric mucosal atrophy in an asymptomatic population with high prevalence of gastric cancer
Rollan, A; Ferreccio, C; Gederlini, A; Serrano, C; Torres, J; Harris, P
World Journal of Gastroenterology, 12(): 7172-7178.

Allergy
Assessing skin prick tests reliability in ECRHS-I
Bousquet, PJ; Chatzi, L; Jarvis, D; Burney, P
Allergy, 63(3): 341-346.
10.1111/j.1398-9995.2007.01581.x
CrossRef
Journal of Psychosomatic Research
Sensitivity and specificity of the Beck Depression Inventory in cardiologic inpatients: How useful is the conventional cut-off score?
Forkmann, T; Vehren, T; Boecker, M; Norra, C; Wirtz, M; Gauggel, S
Journal of Psychosomatic Research, 67(4): 347-352.
10.1016/j.jpsychores.2009.04.003
CrossRef
Journal of Proteome Research
IPO-38 is identified as a novel serum biomarker of gastric cancer based on clinical proteomics technology
Hao, Y; Yu, YY; Wang, LS; Yan, M; Ji, J; Qu, Y; Zhang, J; Liu, BY; Zhu, ZG
Journal of Proteome Research, 7(9): 3668-3677.
10.1021/pr700638k
CrossRef
American Journal of Emergency Medicine
Prognostic factors for primary septicemia and wound infection caused by Vibrio vulnificus
Chou, TNK; Lee, YT; Lai, YY; Chao, WN; Yang, C; Chen, CC; Wang, PH; Lin, DB; Wong, RH; Chen, SC
American Journal of Emergency Medicine, 28(4): 424-431.
10.1016/j.ajem.2008.12.037
CrossRef
Clinical Chemistry
Bias in sensitivity and specificity caused by data-driven selection of optimal cutoff values: Mechanisms, magnitude, and solutions
Leeflang, MMG; Moons, KGM; Reitsma, JB; Zwinderman, AH
Clinical Chemistry, 54(4): 729-737.
10.1373/clinchem.2007.096032
CrossRef
Age and Ageing
The comparative ability of eight functional mobility tests for predicting falls in community-dwelling older people
Tiedemann, A; Shimada, H; Sherrington, C; Murray, S; Lord, S
Age and Ageing, 37(4): 430-435.
10.1093/ageing/afn100
CrossRef
American Journal of Epidemiology
Re: "The inconsistency of 'optimal' cutpoints obtained using two criteria based on the receiver operating characteristic curve" - The authors reply
Perkins, NJ; Schisterman, EF
American Journal of Epidemiology, 164(7): 708.
10.1093/aje/kwj293
CrossRef
Biostatistics
Pooling biospecimens and limits of detection: effects on ROC curve analysis
Mumford, SL; Schisterman, EF; Vexler, A; Liu, AY
Biostatistics, 7(4): 585-598.
10.1093/biostatistics/kxj027
CrossRef
Veterinary Immunology and Immunopathology
Evanescent-wave biosensor for field serodiagnosis of tortoise mycoplasmosis
Brown, DR; Wendland, LD; Ortiz, GJ; Kramer, MF; Lim, DV; Brown, MB; Klein, PA
Veterinary Immunology and Immunopathology, 124(): 322-331.
10.1016/j.vetimm.2008.04.009
CrossRef
Pediatric Neurology
Critical score of Glasgow Coma Scale for pediatric traumatic brain injury
Chung, CY; Chen, CL; Cheng, PT; See, LC; Tang, SFT; Wong, AMK
Pediatric Neurology, 34(5): 379-387.
10.1016/j.pediatrneurol.2005.10.012
CrossRef
Cardiovascular Diabetology
Does the association of the triglyceride to high-density lipoprotein cholesterol ratio with fasting serum insulin differ by race/ethnicity?
Li, CY; Ford, ES; Meng, YX; Mokdad, AH; Reaven, GM
Cardiovascular Diabetology, 7(): -.
ARTN 4
CrossRef
Clinical and Vaccine Immunology
Improved enzyme-linked immunosorbent assay to reveal Mycoplasma agassizii exposure: a valuable tool in the management of environmentally sensitive tortoise populations
Wendland, LD; Zacher, LA; Klein, PA; Brown, DR; Demcovitz, D; Littell, R; Brown, MB
Clinical and Vaccine Immunology, 14(9): 1190-1195.
10.1128/CVI.00108-07
CrossRef
Clinical Linguistics & Phonetics
Relation of melody complexity in infants' cries to language outcome in the second year of life: A longitudinal study
Wermke, K; Leising, D; Stellzig-Eisenhauer, A
Clinical Linguistics & Phonetics, 21(): 961-973.
10.1080/02699200701659243
CrossRef
Communications in Statistics-Simulation and Computation
Confidence intervals for the Youden index and corresponding optimal cut-point
Schisterman, EF; Perkins, N
Communications in Statistics-Simulation and Computation, 36(3): 549-563.
10.1080/03610910701212181
CrossRef
Bmc Medical Research Methodology
oA novel nonparametric approach for estimating cut-offs in continuous risk indicators with application to diabetes epidemiology
Klotsche, J; Ferger, D; Pieper, L; Rehm, J; Wittchen, HU
Bmc Medical Research Methodology, 9(): -.
ARTN 63
CrossRef
Computers in Biology and Medicine
Statistical estimation of diagnosis with genetic markers based on decision tree analysis of complex disease
Liu, H; Gai, LP
Computers in Biology and Medicine, 39(): 989-992.
10.1016/j.compbiomed.2009.07.015
CrossRef
Leukemia & Lymphoma
Prognostic immunophenotypic biomarker studies in diffuse large B cell lymphoma with special emphasis on rational determination of cut-off scores
Tzankov, A; Zlobec, I; Went, P; Robl, H; Hoeller, S; Dirnhofer, S
Leukemia & Lymphoma, 51(2): 199-212.
10.3109/10428190903370338
CrossRef
Journal of Clinical Epidemiology
Maximization of the sum of sensitivity and specificity as a diagnostic cutpoint criterion
Kaivanto, K
Journal of Clinical Epidemiology, 61(5): 517-518.
10.1016/j.jclinepi.2007.10.011
CrossRef
Journal of Behavior Therapy and Experimental Psychiatry
Development and validation of the Rasch-based depression screening (DESC) using Rasch analysis and structural equation modelling
Forkmann, T; Boecker, M; Wirtz, M; Eberle, N; Westhofen, M; Schauerte, P; Mischke, K; Kircher, T; Gauggel, S; Norra, C
Journal of Behavior Therapy and Experimental Psychiatry, 40(3): 468-478.
10.1016/j.jbtep.2009.06.003
CrossRef
Journal of Nonparametric Statistics
Nonparametric deconvolution of density estimation based on observed sums
Vexler, A; Liu, AY; Schisterman, E
Journal of Nonparametric Statistics, 22(1): 23-39.
10.1080/10485250903094286
CrossRef
Clinical Chemistry
Heterogeneous Nuclear Ribonucleoprotein H1, a Novel Nuclear Autoantigen
Van den Bergh, K; Hooijkaas, H; Blockmans, D; Westhovens, R; De Beeck, KO; Verschueren, P; Dufour, D; Van De Merwe, JP; Fijak, M; Klug, J; Michiels, G; Devogelaere, B; De Smedt, H; Derua, R; Waelkens, E; Blanckaert, N; Bossuyt, X
Clinical Chemistry, 55(5): 946-954.
10.1373/clinchem.2008.115626
CrossRef
Clinical Cancer Research
Association of pretherapeutic expression of chemotherapy-related genes with response to neoadjuvant chemotherapy in Barrett carcinoma
Langer, R; Specht, K; Becker, K; Ewald, P; Bekesch, M; Sarbia, M; Busch, R; Feith, M; Stein, H; Siewert, JR; Hofler, H
Clinical Cancer Research, 11(): 7462-7469.
10.1158/1078-0432.CCR-05-0042
CrossRef
American Journal of Epidemiology
The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve
Perkins, NJ; Schisterman, EF
American Journal of Epidemiology, 163(7): 670-675.
10.1093/aje/kwj063
CrossRef
European Journal of Clinical Nutrition
Preferred clinical measures of central obesity for predicting mortality
Welborn, TA; Dhaliwal, SS
European Journal of Clinical Nutrition, 61(): 1373-1379.
10.1038/sj.ejcn.1602656
CrossRef
2007 Ieee International Symposium on Industrial Electronics, Proceedings, Vols 1-8
A fuzzy membership function design methodology based on histogram and ROC curve analyses for broken rotor bar detection
Ayhan, B; Chow, MY; Trussell, HJ
2007 Ieee International Symposium on Industrial Electronics, Proceedings, Vols 1-8, (): 1080-1085.

Statistics in Medicine
Youden Index and the optimal threshold for markers with mass at zero
Schisterman, EF; Faraggi, D; Reiser, B; Hu, J
Statistics in Medicine, 27(2): 297-315.
10.1002/sim.2993
CrossRef
European Journal of Clinical Investigation
Angiogenin outperforms VEGF, EPCs and CECs in predicting Dukes' and AJCC stage in colorectal cancer
Ramcharan, SK; Lip, GYH; Stonelake, PS; Blann, AD
European Journal of Clinical Investigation, 43(8): 801-808.
10.1111/eci.12108
CrossRef
Analytica Chimica Acta
Magneto immunofluorescence assay for diagnosis of celiac disease
Kergaravat, SV; Beltramino, L; Garnero, N; Trotta, L; Wagener, M; Fabiano, SN; Pividori, MI; Hernandez, SR
Analytica Chimica Acta, 798(): 89-96.
10.1016/j.aca.2013.09.009
CrossRef
Bmc Public Health
Determinants of vitamin a deficiency in children between 6 months and 2 years of age in Guinea-Bissau
Danneskiold-Samsoe, N; Fisker, AB; Jorgensen, MJ; Ravn, H; Andersen, A; Balde, ID; Leo-Hansen, C; Rodrigues, A; Aaby, P; Benn, CS
Bmc Public Health, 13(): -.
ARTN 172
CrossRef
Eye
Relevance of TSH-receptor antibody levels in predicting disease course in Graves' orbitopathy: comparison of the third-generation TBII assay and Mc4-TSI bioassay
Jang, SY; Shin, DY; Lee, EJ; Lee, SY; Yoon, JS
Eye, 27(8): 964-971.
10.1038/eye.2013.120
CrossRef
Academic Radiology
Statistical Power Considerations for a Utility Endpoint in Observer Performance Studies
Abbey, CK; Samuelson, FW; Gallas, BD
Academic Radiology, 20(7): 798-806.
10.1016/j.acra.2013.02.008
CrossRef
International Journal of Radiation Oncology Biology Physics
Duodenal Toxicity After Fractionated Chemoradiation for Unresectable Pancreatic Cancer
Kelly, P; Das, P; Pinnix, CC; Beddar, S; Briere, T; Pham, M; Krishnan, S; Delclos, ME; Crane, CH
International Journal of Radiation Oncology Biology Physics, 85(3): E143-E149.
10.1016/j.ijrobp.2012.09.035
CrossRef
Plos One
Can Point-of-Care Urine LAM Strip Testing for Tuberculosis Add Value to Clinical Decision Making in Hospitalised HIV-Infected Persons?
Peter, JG; Theron, G; Dheda, K
Plos One, 8(2): -.
ARTN e54875
CrossRef
Clinical Radiology
Serial ultrasound findings associated with early liver transplantation after Kasai portoenterostomy in biliary atresia
Jeon, TY; Yoo, SY; Kim, JH; Eo, H; Lee, SK
Clinical Radiology, 68(6): 588-594.
10.1016/j.crad.2012.12.004
CrossRef
Manual Therapy
Can innominate motion be used to identify persons with ankylosing spondylitis? A pilot study
Bussey, MD; Milosavljevic, S
Manual Therapy, 18(2): 118-123.
10.1016/j.math.2012.07.010
CrossRef
Critical Care
The prognostic value of left ventricular systolic function measured by tissue Doppler imaging in septic shock
Weng, L; Liu, YT; Du, B; Zhou, JF; Guo, XX; Peng, JM; Hu, XY; Zhang, SY; Fang, Q; Zhu, WL
Critical Care, 16(3): -.
ARTN R71
CrossRef
Biostatistics
Regression models for group testing data with pool dilution effects
McMahan, CS; Tebbs, JM; Bilder, CR
Biostatistics, 14(2): 284-298.
10.1093/biostatistics/kxs045
CrossRef
Teaching and Learning in Medicine
Predicting Performance on the Medical Council of Canada Qualifying Exam Part II
Woloschuk, W; McLaughlin, K; Wright, B
Teaching and Learning in Medicine, 25(3): 237-241.
10.1080/10401334.2013.797351
CrossRef
Anesthesiology
Statistical Evaluation of a Biomarker
Ray, P; Manach, YL; Riou, B; Houle, TT
Anesthesiology, 112(4): 1023-1040.
10.1097/ALN.0b013e3181d47604
PDF (1338) | CrossRef
Critical Care Medicine
Detecting ineffective triggering in the expiratory phase in mechanically ventilated patients based on airway flow and pressure deflection: Feasibility of using a computer algorithm*
Chen, C; Lin, W; Hsu, C; Cheng, K; Lo, C
Critical Care Medicine, 36(2): 455-461.
10.1097/01.CCM.0000299734.34469.D9
PDF (667) | CrossRef
Critical Care Medicine
Prognostic value of plasma N-terminal probrain natriuretic peptide levels in the acute respiratory distress syndrome*
Bajwa, EK; Januzzi, JL; Gong, MN; Thompson, BT; Christiani, DC
Critical Care Medicine, 36(8): 2322-2327.
10.1097/CCM.0b013e318181040d
PDF (591) | CrossRef
European Journal of Cardiovascular Prevention & Rehabilitation
Anthropometric measures and absolute cardiovascular risk estimates in the Australian Diabetes, Obesity and Lifestyle (AusDiab) Study
Chen, L; Peeters, A; Magliano, DJ; Shaw, JE; Welborn, TA; Wolfe, R; Zimmet, PZ; Tonkin, AM
European Journal of Cardiovascular Prevention & Rehabilitation, 14(6): 740-745.
10.1097/HJR.0b013e32816f7739
PDF (96) | CrossRef
Back to Top | Article Outline

Supplemental Digital Content

Back to Top | Article Outline

© 2005 Lippincott Williams & Wilkins, Inc.

Twitter  Facebook

Login

Article Tools

Images

Share