Based on various prevalences of and cut points for defining virologic failure, we compared relative efficiency and negative predictive value for four approaches (individual samples, minipool, minipool + algorithm, and matrix approaches) and various pool sizes within each of the pooling approaches (minipool, minipool + algorithm, and matrix approaches). Relative efficiency was defined as one minus the average number of assays performed divided by the number of samples. We use a different definition of efficiency than others27 because of the natural interpretation that the best algorithm will have the highest values of efficiency relative to individual testing. With this definition, an algorithm with efficiency zero has no advantage over individual testing and an algorithm with efficiency of 0.25 uses 25% fewer samples than would be used with individual testing.
Negative predictive values were compared for the various approaches to identify individual samples that had viral load values below the same range of cut points (500, 1000, and 1500 HIV RNA copies/mL) similar to the comparisons of relative efficiency. For our purposes, the negative predictive value was the proportion of individuals who were virologically suppressed below the cut point defining virologic failure (true- and test-negative) among those who were considered suppressed by each proposed method (test-negative). We focus here on achieving high negative predictive values, because of the clinical importance of missing virologic failure among patients receiving ART by the testing approaches.
Additionally, we evaluated the various approaches in the context of the following clinically relevant factors: 1) level of viral load for defining virologic failure; 2) standard deviation of the viral load assay; and 3) prevalence of virologic failure in the sampled population. Screening approaches for higher prevalence of virologic failure and assays with larger standard deviation were expected to be less efficient than for lower prevalence of virologic failure or assays with smaller standard deviation.
Pool sizes of three, four, five, six, seven, eight, 10, and 20 were considered. Because the lowest possible level of viral load detection depends on the dilution factor of the pool, the level of detection was considered in relation to the size of the pool. For example, with a lower detection limit for a viral load assay of 50 copies/mL and a pool size of 10, the on average lowest level of virologic failure that we can expect to detect for any individual sample based on the pooled analysis was 500 copies/mL. When pools of 20 individual samples were considered, the lowest detectable level of virologic failure was 1000 copies/mL. All simulations were run using the statistical software Stata34 and were repeated 1000 times for all conditions.
Of the three pooling approaches, most demonstrated relatively high efficiency (greater than 0.70) when the prevalence of virologic failure was low (less than 3%) (Fig. 3), that is, less than 30% of viral load assays would be used in a pooling approach relative to performing viral loads for each individual sample. When the prevalence of virologic failure was higher (greater than 3%), relative efficiency varied markedly; Dorfman's18 minipools showed lower relative efficiency than algorithms that used the quantitative information available. A matrix approach with intermediate pool size (eight or 10) appeared to be the most efficient approach when: 1) virologic failure was defined as 1500 copies/mL or greater; 2) the standard deviation of the assay was 0.12; and 3) the prevalence of virologic failure of the sampled population was between 5% and 20%. Specifically, the relative efficiency for the matrix approach with a pool size of 10 ranged from 0.68 to 0.33 and for a matrix approach with a pool size of eight, it ranged from 0.65 to 0.32 when the prevalence of virologic failure was 5% and 20%, respectively. Among the approaches that use the minipools + algorithm, the one with five samples per pool appeared most efficient under similar conditions with relative efficiency from 0.63 to 0.26 for a prevalence of virologic failure of 5% and 20%, respectively. Thus, the relative efficiency of the five minipool + algorithm was close (no more than 0.07 difference) to the most efficient matrix approaches. For minipool and minipool + algorithm approaches of the same size, there was an advantage for using the quantitative information when the prevalence of virologic failure was higher than 5%. According to Dorfman's studies,18 the optimal pool size in the minipool approach is five for a prevalence of 7%, which can achieve a 50% reduction in the number of samples being tested. For our setting, the minipool of size five achieved a 50% reduction in samples being tested for a prevalence of 6%. When quantitative information was used (minipool + algorithm), the same reduction could be achieved for a prevalence of 9% and with the matrix approach of size 10, the same reduction could be achieved for a prevalence of 11%. Similarly, an efficiency of 30% can be achieved for a prevalence of 11%, 18%, and 21% for the minipool, minipool + algorithm, and matrix approaches, respectively. As a result, the same efficiency can be achieved at approximately double the prevalence when using quantitative information and the matrix approach compared with when quantitative information is ignored like in the minipool approach. The relative efficiency over the various approaches changed only slightly when the standard deviation of the assay was 0.20. For example, when the prevalence of virologic failure was 5% in the sampled population, the relative efficiency for the matrix approach with a pool size of 10 was: 1) 0.66 when the standard deviation of the viral load assay was 0.20; 2) 0.68 when the standard deviation was 0.12; and 3) 0.73 when the standard deviation was zero. For Dorfman's minipool approach, the most efficient pool size was three (compare Table 1 of Dorfman18) with relative efficiency ranging from 0.52 to 0.13 when the prevalence of virologic failure was 5% and 20%, respectively.
Overall, pool sizes of three and four had the highest relative efficiency among the minipool and minipool + algorithm approaches and a pool size of 10 had the highest relative efficiency among the matrix approaches when the prevalence of virologic failure was between 5% and 20% (Fig. 4). Results were not substantially different when virologic failure was defined as 500 or greater, 1000 copies/mL or greater, or 1500 copies/mL or greater. For example, when using the matrix approach with a pool size of 10, assay standard deviation of 0.12, and a prevalence of virologic failure of 20%, the relative efficiency was 0.68 when virologic failure was defined at 1500 copies/mL or greater, 0.66 at 1000 copies/mL or greater, and 0.65 at 500 copies/mL or greater.
Negative Predictive Values
The results for negative predictive values were uniformly good. For a prevalence of virologic failure up to 10%, negative predictive values were at least 99% and 97% for the matrix approach with assay standard deviations of 0.12 and 0.20 copies/mL, respectively. Similarly, the corresponding negative predictive values were 99% and 98% for all minipool + algorithm approaches. For a prevalence of virologic failure up to 20%, negative predictive values were always at least 92% for all matrix approaches and always above 96% for the minipool + algorithm approach when the standard deviation of the assay was 0.12. These results changed only minimally for an assay standard deviation of 0.20. Negative predictive values were always higher than 99% for Dorfman's minipool approach (without algorithm). These results remained virtually unchanged for different definitions of virologic failure (500, 1000, and 1500 HIV RNA copies/mL).
In addition to relative efficiency and negative predictive value, there are other factors that can influence the clinical usefulness of these approaches. One is the local need of turnaround time of the results, which will depend on availability of the assays, personnel performing the assays, and size of the clinical population receiving ART. The proposed pooling strategies will vary in the time from sample collection to the time individual viral load results are available to the clinician for management of their patients. For example, if a laboratory has the capability of performing 20 viral loads a day, then 20 individuals could be screened daily using individual viral load testing and it would take 5 days to screen 100 individuals. If minipools of size five are used, then 100 individuals could be screened in the first day, but some pools would require additional testing and, therefore, additional days would be required before results are available. The number of additional days would depend on the prevalence of virologic failure in the sampled population and the specific pooling approach. When 100 patients are screened using minipools + algorithm of size five, virologic failure is defined as 1500 HIV RNA copies/mL or greater and the prevalence of virologic failure is 10%, there would be an average of 2 days until the viral loads of all samples have been resolved. Similarly, the initial screening of the same 100 patients using the matrix approach with a pool size of 10 would require 1 day, but the complete resolution of all samples would require on average 28 days (1 day for testing all row and column pools and an average of approximately 27 days for 27 individual assays that must be tested consecutively); however, on average, 65% of the individual samples would have been resolved on the first day. From a financial perspective, when 100 patients are screened, virologic failure is defined as 1500 HIV RNA copies/mL or greater, assay standard deviation is 0.12, the prevalence of virologic failure is 10%, and one assay costs $75 (as an example), then the average cost per patient tested would be $75 for individual testing and approximately $49, $38, and $35 for using the minipool of size five, minipool + algorithm of size five, and 10 by 10 matrix approaches, respectively, to screen 100 patients.
Because commercial viral load assays cost between U.S. $50 and $150 per test (diagnostics pricing; Clinton Foundation; available at: http://www.clintonfoundation.org/cf-pgm-hs-ai-work3.htm; accessed May 23, 2008),8 the costs of virologic monitoring during ART may exceed those of ART itself, but the development and transmission of drug-resistant HIV may ultimately compromise the effectiveness of ART in many populations.9,31,32,35,36 In resource-constrained settings, less expensive methods to monitor viral replication during ART are necessary to make such monitoring feasible and ART sustainable. In this report, we demonstrate how nucleic acid testing on pooled blood samples can be used to reduce the overall number of viral load tests needed to screen patients receiving ART compared with individual testing. By taking advantage of the quantitative information available, efficiency can be achieved for approximately double the prevalence than for methods that use qualitative information only (eg, disease/no disease). To our knowledge, none of the previous work on pooling methods has exploited quantitative information.
Because the prevalence of virologic failure in the sampled population will impact the relative efficiency and accuracy of the presented pooling and testing methods, other strategies that can identify individuals with virologic failure before nucleic acid testing could greatly increase the usefulness of the proposed methods. Such strategies include the measurement of adherence to ART4,37-39 or longitudinal trajectory of CD4 counts4; however, these methods are by no means perfect predictors of virologic failure4-8 and will need to be evaluated in individual clinical settings. Additionally, the definition of virologic failure is an obvious factor in determining the prevalence of virologic failure in a population. Although some research considers the consequences of different definitions of virologic failure,40-42 it remains unclear which level of HIV RNA viral load constitutes the most important clinical cut point to define virologic failure. Differences regarding negative predictive values among different definitions of virologic failure (500, 1000, and 1500 HIV RNA copies/mL) appear to be very small (2% or less) and reasonably small (7% or less) for prevalence of virologic failure up to 10% and 20%, respectively. Differences in relative efficiency for different definitions of virologic failure (500, 1000, and 1500 HIV RNA copies/mL) were also small. In essence, results regarding relative efficiency and negative predictive value did not depend on the definition of virologic failure. It might be possible to further improve the efficiency of the matrix approach. The current approach for using the quantitative information is simple and straightforward, but more sophisticated methods, for example, for choosing the order in which individual samples are tested, might further improve efficiency.
Because the goal of virologic monitoring during ART is to achieve and maintain a low prevalence of virologic failure in the population of patients receiving ART, another option to achieve low prevalence would be to monitor the population for virologic failure at shorter intervals. Recent HIV treatment guidelines recommend monitoring viral loads every 3 to 4 months1; however, these recommendations are based solely on experts' interpretation of the published literature,1 and more frequent monitoring has been proposed based on prospective clinical trial data.2 More likely, the frequency of virologic monitoring needed to optimize clinical outcomes will vary by clinical setting, which would include factors such as the potency, tolerability, and durability of the available ART regimens; support available for patient adherence; and prevalence of transmitted drug resistance. Taken together, the methods used to monitor for virologic failure during ART will need to be evaluated based on the needs of each clinical setting. We have focused here on relative efficiency and negative predictive value, but depending on the setting, the number of false-positive results might be considered as well. Associated costs could offset some or all of any efficiency achieved through the pooling, like unnecessary resistance testing performed or patients being unnecessarily switched to a new regimen.
Similarly, each clinical program will most likely have unique requirements for assay characteristics (overall costs, turnaround time of results, and monitoring accuracy). For example, in some areas, the overall cost of performing a viral load may be the greatest limiting factor for monitoring for virologic failure; therefore, a clinical program in this setting might best be served by a method that has the highest relative efficiency for the assays independent of the turnaround time of individual results or maximal accuracy. Alternatively, other programs might require more rapid turnaround times for obtaining viral load results such as programs with smaller patient populations receiving ART. These settings would require more time to obtain enough samples to constitute a pool and more frequent testing or smaller pool sizes or even individual viral load testing might be the most cost-effective. Another factor that must be considered for each clinical setting is quality assurance, which will mostly depend on the expertise of the personnel performing the assays (sample handling, processing, and technical consistency with the viral load assay), which could lead to errors in pooling, resolution testing, and inconsistency in calculating viral load results. To further address quality assurance, retesting of samples that are assumed or estimated to be virologically suppressed can be performed using pooling techniques in an efficient manner.43 Similar to choosing a method to screen for virologic failure, the extent and nature of technical training required to perform various pooling approaches and maximize quality assurance will need to be evaluated in each local laboratory and clinical setting.
The proposed approaches are not limited to HIV research but could be used in other settings where quantitative measures are available for screening. In settings where confidentiality is essential, the proposed methods could be applied in two different ways. First, the methods could be used to estimate prevalence.19 Second, if individual identification of samples is important, the estimate of prevalence could be used to guide the choice of pooling method.
We present characteristics for a variety of algorithms and pool sizes that can incorporate available quantitative viral load data and data on how pooling methods can be used efficiently and accurately to monitor for virologic failure in patients receiving ART. These data could be used by individual laboratory and clinical settings to make choices about optimal local virologic monitoring strategies. They may also be used to design efficient pooling strategies for other settings where screening involves quantitative measures. Although promising, further investigation in resource-constrained settings is required to determine if these methods are feasible and cost-effective with respect to the factors that could not be included in the simulations like turnaround time of results, additional personnel costs in local settings, and individual and public health costs of a patient population with virologic failure during ART.
We thank two reviewers and Drs. Susan Little, Robert Schooley, and Matthew C. Strain for insightful comments.
1. Hammer SM, Saag MS, Schechter M, et al. Treatment for adult HIV infection: 2006 recommendations of the International AIDS
Society-USA panel. JAMA
2. Haubrich RH, Currier JS, Forthal DN, et al. A randomized study of the utility of human immunodeficiency virus RNA measurement for the management of antiretroviral therapy. Clin Infect Dis
3. Hughes MD, Johnson VA, Hirsch MS, et al. Monitoring plasma HIV-1 RNA levels in addition to CD4+ lymphocyte count improves assessment of antiretroviral therapeutic response. ACTG 241 Protocol Virology Substudy Team. Ann Intern Med
4. Bisson GP, Gross R, Bellamy S, et al. Pharmacy refill adherence compared with CD4 count changes for monitoring HIV-infected adults on antiretroviral therapy. PLoS Med
5. Petti CA, Polage CR, Quinn TC, et al. Laboratory medicine in Africa: a barrier to effective health care. Clin Infect Dis
6. Bisson GP, Gross R, Strom JB, et al. Diagnostic accuracy of CD4 cell count increase for virologic response after initiating highly active antiretroviral therapy. AIDS
7. Moore DM, Mermin J, Awor A, et al. Performance of immunologic responses in predicting viral load suppression: implications for monitoring patients in resource-limited settings. J Acquir Immune Defic Syndr
8. Fiscus SA, Cheng B, Crowe SM, et al. HIV-1 viral load assays for resource-limited settings. PLoS Med
9. Miller V, Larder BA. Mutational patterns in the HIV genome and cross-resistance following nucleoside and nucleotide analogue drug exposure. Antivir Ther
. 2001;6(Suppl 3):25-44.
10. Phillips AN, Pillay D, Miners AH, et al. Outcomes from monitoring of patients on antiretroviral therapy in resource-limited settings with viral load, CD4 cell count, or clinical observation alone: a computer simulation model. Lancet
11. Calmy A, Ford N, Hirschel B, et al. HIV viral load monitoring in resource-limited regions: optional or necessary? Clin Infect Dis
12. Smith DM, Schooley RT. Running with scissors: using antiretroviral therapy without monitoring viral load. Clin Infect Dis
13. Durant J, Clevenbergh P, Halfon P, et al. Drug-resistance genotyping in HIV-1 therapy: the VIRADAPT randomised controlled trial. Lancet
14. Pilcher CD, McPherson JT, Leone PA, et al. Real-time, universal screening for acute HIV infection in a routine HIV counseling and testing population. JAMA
15. Pilcher CD, Fiscus SA, Nguyen TQ, et al. Detection of acute infections during HIV testing in North Carolina. N Engl J Med
16. Busch MP, Glynn SA, Stramer SL, et al. A new strategy for estimating risks of transfusion-transmitted viral infections based on rates of detection of recently infected donors. Transfusion
17. Patterson KB, Leone PA, Fiscus SA, et al. Frequent detection of acute HIV infection in pregnant women. AIDS
18. Dorfman R. The detection of defective members of large populations. Annals of Mathematical Statistics
19. Hammick PA, Gastwirth JL. Group-testing for sensitive characteristics: extension to higher prevalence levels. International Statistical Review
20. Brookmeyer R. Analysis of multistage pooling studies of biological specimens for estimating disease incidence and prevalence. Biometrics
21. Behets F, Bertozzi S, Kasali M, et al. Successful use of pooled sera to determine HIV-1 seroprevalence in Zaire with development of cost-efficiency
22. Cahoon-Young B, Chandler A, Livermore T, et al. Sensitivity and specificity of pooled versus individual sera in a human immunodeficiency virus antibody prevalence study. J Clin Microbiol
23. Gastwirth JL, Hammick PA. Estimation of the prevalence of a rare disease, preserving the anonymity of the subjects by group-testing-application to Estimating the prevalence of AIDS
antibodies in blood-donors. Journal of Statistical Planning and Inference
24. Tu XM, Litvak E, Pagano M. On the Informativeness and accuracy of pooled testing in estimating prevalence of a rare disease-application to HIV screening. Biometrika
25. Quinn TC, Brookmeyer R, Kline R, et al. Feasibility of pooling sera for HIV-1 viral RNA to diagnose acute primary HIV-1 infection and estimate HIV incidence. AIDS
26. Kline RL, Brothers TA, Brookmeyer R, et al. Evaluation of human immunodeficiency virus seroprevalence in population surveys using pooled sera. J Clin Microbiol
27. Westreich DJ, Hudgens MG, Fiscus SA, et al. Optimizing screening for acute HIV infection with pooled nucleic acid amplification tests. J Clin Microbiol
28. Kim HY, Hudgens MG, Dreyfuss JM, et al. Comparison of group testing algorithms for case identification in the presence of test error. Biometrics
29. Brambilla D, Reichelderfer PS, Bremer JW, et al. The contribution of assay variation and biological variation to the total variability of plasma HIV-1 RNA measurements. The Women Infant Transmission Study Clinics. Virology Quality Assurance Program. AIDS
30. Jagodzinski LL, Wiggins DL, McManis JL, et al. Use of calibrated viral load standards for group M subtypes of human immunodeficiency virus type 1 to assess the performance of viral RNA quantitation tests. J Clin Microbiol
31. Harrigan PR, Hogg RS, Dong WW, et al. Predictors of HIV drug-resistance mutations in a large antiretroviral-naive cohort initiating triple antiretroviral therapy. J Infect Dis
32. Haupts S, Ledergerber B, Boni J, et al. Impact of genotypic resistance testing on selection of salvage regimen in clinical practice. Antivir Ther
33. Hirsch MS, Brun-Vezinet F, Clotet B, et al. Antiretroviral drug resistance testing in adults infected with human immunodeficiency virus type 1: 2003 recommendations of an International AIDS
Society-USA Panel. Clin Infect Dis
34. StataCorp. Stata Statistical Software: Release 10
. College Station, TX: StataCorp; 2007.
35. Little SJ, Holte S, Routy JP, et al. Antiretroviral-drug resistance among patients recently infected with HIV. N Engl J Med
36. Vijayaraghavan A, Efrusy MB, Mazonson PD, et al. Cost-effectiveness of alternative strategies for initiating and monitoring highly active antiretroviral therapy in the developing world. J Acquir Immune Defic Syndr
37. Haubrich RH, Little SJ, Currier JS, et al. The value of patient-reported adherence to antiretroviral therapy in predicting virologic and immunologic response. California Collaborative Treatment Group. AIDS
38. Gifford AL, Bormann JE, Shively MJ, et al. Predictors of self-reported adherence and plasma HIV concentrations in patients on multidrug antiretroviral regimens. J Acquir Immune Defic Syndr
39. Paterson DL, Swindells S, Mohr J, et al. Adherence to protease inhibitor therapy and outcomes in patients with HIV infection. Ann Intern Med
40. Raboud JM, Seminari E, Rae SL, et al. Comparison of costs of strategies for measuring levels of human immunodeficiency virus type 1 RNA in plasma by using Amplicor and Ultra Direct assays. J Clin Microbiol
41. Raboud JM, Rae S, Hogg RS, et al. Suppression of plasma virus load below the detection limit of a human immunodeficiency virus kit is associated with longer virologic response than suppression below the limit of quantitation. J Infect Dis
42. Macias J, Palomares JC, Mira JA, et al. Transient rebounds of HIV plasma viremia are associated with the emergence of drug resistance mutations in patients on highly active antiretroviral therapy. J Infect
43. Johnson WO, Gastwirth JL. Dual group screening. Journal of Statistical Planning and Inference
Keywords:© 2010 Lippincott Williams & Wilkins, Inc.
AIDS; efficiency; matrix