Performance of Commercial Herpes Simplex Virus Type-2 Antibody Tests Using Serum Samples From Sub-Saharan Africa: A Systematic Review and Meta-analysis

Abstract

Background: 

Several commercial type-specific serologic tests are available for herpes simplex virus type 2 (HSV-2). Poor specificity of some tests has been reported on samples from sub-Saharan Africa.

Methods: 

To summarize the performance of the tests using samples from sub-Saharan Africa, we conducted a systematic review of publications reporting performance of commercially available HSV-2 tests against a gold standard (Western Blot or monoclonal antibody-blocking EIA). We used random-effects meta-analyses to summarize sensitivity and specificity of the 2 most commonly evaluated tests, Kalon gG2 enzyme-linked immunosorbent assay (ELISA), and Focus HerpeSelect HSV-2 ELISA.

Results: 

We identified 10 eligible articles that included 21 studies of the performance of Focus, and 12 of Kalon. The primary analyses included studies using the manufacturers' cut-offs (index value = 1.1). Focus had high sensitivity (random effects summary estimate 99%, 95% confidence interval [CI]: 99%–100%) but low specificity (69%, 95% CI: 59%–80%). Kalon had sensitivity of 95% (95% CI: 93%–97%) and specificity of 91% (95% CI: 86%–95%). Specificity of Focus was significantly lower (P = 0.002) among HIV-positive (54%, 95% CI: 40%–68%) than HIV-negative individuals (69%, 95% CI: 56%–82%). When the cut-off optical density index was increased above the recommended value of 1.1 to between 2.2 and 3.5, the specificity of Focus increased to 85% (95% CI: 77%–92%).

Conclusions: 

Sensitivity and specificity of HSV-2 tests used in sub-Saharan Africa vary by setting, and are lower than reported from studies in the United States and Europe. Increasing the cut-off optical density index may improve test performance. Evaluation of test performance in a given setting may help deciding which test is most appropriate.

In Brief

A systematic review and meta-analysis of the performance of herpes simplex virus type 2 serological tests in Africa found that Kalon gG2 enzyme-linked immunosorbent assay performed more favorably than Focus HerpeSelect herpes simplex virus type 2 enzyme-linked immunosorbent assay. Increasing the cut-off optical density improved performance.

Herpes simplex virus type 2 (HSV-2) infection is the most common cause of genital ulcer disease worldwide,^1,2 and is characterized by asymptomatic periods during which subclinical genital shedding may frequently occur.^3,4 Accurate serological tests are therefore needed to determine HSV-2 infection status rather than relying on presence of symptoms. HSV-2 is associated with increased risk of HIV acquisition and infectiousness,⁵ and epidemiologic studies of the association with HIV rely on accurate laboratory tests to identify HSV-2 serostatus. For example, several recent randomized controlled trials have evaluated the potential for HSV suppressive therapy to reduce HIV acquisition, infectivity, and transmission^6–13 and rely on high test specificity to avoid enrolment of HSV-2 uninfected individuals. In addition, studies using HSV-2 as a biologic marker for risky sexual behavior require accurate tests for HSV-2 infection.^14,15 Accurate tests for HSV-2 are also important for determining prevalence or incidence in the general population as might be the case in studies evaluating risk factors for HIV or in studies evaluating a vaccine for HSV-2, if this were available.

Gold standard noncommercial tests for HSV-2 include the immunodot enzyme assay (IEA) developed at Emory University, Atlanta, GA; the Western blot test developed at the University of Washington (UW-WB) and the monoclonal antibody-blocking enzyme immunoassay (MAb EIA) developed by the Central Public Health Laboratory in the United Kingdom.¹⁶ These tests are used in their respective reference laboratories but are not feasible to replicate in many settings, thereby limiting their suitability for large-scale epidemiologic studies. The UW-WB test has been used as a gold standard in several studies, including evaluation of commercial serological assays required for clearance by the Food and Drug Administration (United States), and in the evaluation of the performance of other gold standard tests.^16,17 The IEA uses immunoaffinity purified glycoprotein gG-2, which is specific for HSV-2.^18,19 When evaluating IEA in symptomatic patients with culture confirmed shedding of HSV-2, it showed a sensitivity of 98% and specificity of 99%¹⁸ and has been used in large national studies to document epidemiologic trends of HSV-2 in the United States.^18,19 The MAb EIA is the main reference test for HSV-2 in the United Kingdom.¹⁶ The performance of MAb EIA was found to be comparable to UW-WB among samples from England and Wales (98% concordant results among 100 samples tested by both assays)²⁰ and it performed well when compared against UW-WB in African samples (sensitivity = 98%, specificity = 97%).¹⁷

Several type-specific serological tests for HSV-2 are commercially available. These tests detect the HSV-2 specific glycoprotein gG-2, and therefore, can distinguish between HSV-1 (gG-1) and HSV-2.¹⁶ This distinction is essential given the almost universal prevalence of HSV-1 infection in many settings.²¹ These commercial tests are easier, faster, and more cost-effective to perform than Western blot (WB). Studies from industrialized countries indicate good overall performance,²² but poor specificity has been noted in several studies from sub-Saharan Africa.^17,22,23 We conducted a systematic review of studies evaluating the most commonly used commercial HSV-2 serologic tests against accepted gold standards in African populations and a meta-analysis of test performance, including effect-modification by HIV infection.

MATERIALS AND METHODS

We searched PubMed up to February 7, 2009, to identify studies evaluating the performance of HSV-2 serologic tests against accepted gold standards (WB or MAb-EIA) in sub-Saharan African populations. An initial exploratory search was done using the phrases “HSV-2 test performance” or “HSV-2 test validation” and following links for “related articles.” Reference lists of these articles were checked for other relevant studies. The MeSH terms of these relevant articles were then examined and a systematic review was performed using the following combinations of MeSH terms: (“Herpesvirus 2, Human” OR “Herpes Genitalis”) and (“Laboratory Techniques and Procedures” OR “Reagent Kits, Diagnostic”). The reference lists of identified studies were also searched for additional studies. In addition, we obtained known unpublished work and draft/in press articles from authors. The data abstracted were descriptive items (author name, journal, publication year), gold standard used, HSV-2 test, and cut-off used, and sensitivity and specificity. Further details were obtained from authors as necessary. Many publications included data of test performance stratified by HIV status, and these were used for separate analyses.

Studies were eligible for inclusion in the meta-analysis for the following reasons: (i) whether they evaluated performance of either Kalon or Focus using WB or MAb-EIA as the gold standard on blood samples from sub-Saharan Africa, (ii) whether they reported sensitivity and/or specificity or whether they provided sufficient data to calculate these, and (iii) whether they reported the cut-off optical density (OD) index value used.

To pool estimates of sensitivity and specificity across studies, we conducted a random-effects meta-analysis using Stata version 10.0. The primary analysis included only studies that assessed test performance using the manufacturer's recommended cut-off OD index value. Confidence intervals (95% CI) were estimated from the data using the Wilson score interval, which is suitable for probabilities approaching 1.²⁴ A further analysis was restricted to studies that used OD cut-off values other than that recommended by the manufacturers. Possible heterogeneity due to HIV infection was explored using meta-regression. In 1 article, analyses were based on MAb EIA as the gold standard, because this comparison had more complete data and provided results similar to those of the UW-WB test.¹⁷

RESULTS

Of 882 abstracts identified through the literature search, 42 were deemed potentially relevant as the abstract indicated they were likely to fulfill the eligibility criteria. Full-text copies of these articles were reviewed, and 7 of these were found to be relevant.^{17,22,23,25–28} In addition, one recently published article was included that was identified during the initial exploratory search but did not appear in the final search as it had not yet been indexed,²⁹ 2 draft papers (now in press) were obtained from collaborating coauthors (Lingappa et al., personal communication, 2009).³⁰ Data from one unpublished evaluation of the performance of Kalon conducted by the MRC/UVRI Uganda Research Unit on AIDS in the general population of Masaka district, Uganda, were also included. We excluded one article²⁶ from further analyses as it used data from the same population from Rakai, Uganda, as a more recent study.²³ Altogether these 10 articles contained 35 substudies from 18 study populations from 8 African countries.

The most commonly evaluated assays were the Focus HerpeSelect HSV-2 enzyme-linked immunosorbent assay (ELISA) (Focus Technologies, Cypress, CA) (21 substudies) and the Kalon gG2 ELISA (Kalon Biologicals Ltd, Guilford, United Kingdom) (12 substudies). In addition, 2 substudies evaluated the performance of Biokit Rapid Assay (BioKit USA Inc, Lexington, MA)²³ (Lingappa et al., personal communication) (Table 1).

TABLE 1: Performance of Herpes Simplex Virus Type-2 (HSV-2) Serological Assays in African Populations

TABLE 1: (Continued)

The pooled sensitivity and specificity of Focus and Kalon are shown in Figure 1. Performance of Focus using the manufacturer's recommended cut-off OD index value of 1.1 was assessed in 17 study populations^{17,22,23,25,27–30} (Lingappa et al., personal communication) adding up to 3387 samples. The random-effects summary sensitivity for Focus was 99% (95% CI: 99–100), but specificity was poor (69%, 95% CI: 59–80). There was significant between-study heterogeneity for specificity (P < 0.001). Performance of Kalon using the manufacturer's recommended cut-off OD index value of 1.1 was assessed in 9 study populations^{17,23,28–30} (Lingappa et al., personal communication) and in the unpublished evaluation in Masaka district, Uganda, adding to 2206 samples. The summary estimate of sensitivity was 95% (95% CI: 93–97) with specificity of 91% (95% CI: 86–95). Specificity results from 1 study (Hogrefe, Kenya B) were excluded, because there was only 1 negative result on UW-WB and no negative result on Focus. For Kalon, there was significant between-study heterogeneity for both sensitivity (P = 0.005) and specificity (P < 0.001).

Figure 1.: Sensitivity and specificity of Focus and Kalon HSV-2 serological tests in African populations at manufacturer's recommended OD cut-offs.

Four studies evaluated Focus using a higher OD cut-off (between 2.2 and 3.5) totaling 1530 samples^22,23,29 (Lingappa et al., personal communication) (Table 1). Amongst these studies, the sensitivity was 91% (95% CI: 85%–97%) and specificity 85% (95% CI: 79%–92%) (Fig. 2). Similarly, 3 studies evaluated Kalon using a higher OD cut-off for a total of 923 samples^23,29 including the unpublished Masaka study (Table 1). The pooled estimate of sensitivity was 93% (95% CI: 90%–95%) and specificity 89% (95% CI: 83%–95%). At these higher cut-off ODs also, there was significant heterogeneity for both sensitivity (P < 0.001) and specificity (P = 0.004) for Focus, and for specificity of Kalon (P = 0.03).

Figure 2.: Sensitivity and specificity of Focus and Kalon HSV-2 serological tests at higher OD cut-off. Higher cut-off for Focus: Ashley-Morrow, Nigeria = 3.5, Gamiel, Uganda = 3.2, Smith, Kenya = 3.5, Lingappa, Uganda = 2.2. Higher cut-off for Kalon: Gamiel, Uganda = 1.5, Smith, Kenya = 1.2, MRC, Uganda = 1.5.

Using the manufacturer's recommended cut-off OD index value of 1.1, sensitivity of both Focus and Kalon were high and similar in HIV-positive and HIV-negative samples (Fig. 3). In contrast, specificity of Focus was significantly lower among HIV-positive (54%, 95% CI: 40–68) than among HIV-negative samples (69%, 95% CI: 56–82; P = 0.02) (Fig. 4), although there was substantial between-study heterogeneity amongst HIV-negative samples (P < 0.001). There was a similar trend for Kalon, although the differences by HIV status were less pronounced. Among HIV-positive samples, specificity was 88% (95% CI: 75–100); among HIV-negative samples, specificity was 93% (95% CI: 88–98) (P = 0.7). However, there was substantial heterogeneity for specificity of Kalon among both the HIV-positive (P = 0.001) and HIV-negative samples (P < 0.001). Of note, few HIV-positive samples were also HSV-2 seronegative, so there were wide confidence intervals for specificity of the tests among HIV-positive samples.

Figure 3.: Sensitivity of Focus and Kalon HSV-2 serological tests by HIV status in African populations.

Figure 4.: Specificity of Focus and Kalon HSV-2 serological tests by HIV status in African populations.

The performance of Biokit was inconsistent in the 2 studies that evaluated it in Uganda. Biokit had a sensitivity of 95.8% and specificity of 56.1% in Rakai District, and a sensitivity of 86.4% and specificity of 97% in Kampala city (Lingappa et al., personal communication).²³

DISCUSSION

Most studies of HSV-2 epidemiology in sub-Saharan Africa have used the Kalon or Focus assays. Our findings show that Focus tended to have low specificity (summary estimate 69%) compared with the gold standard when using the manufacturer's recommended cut-off (OD = 1.1). Kalon tends to perform better when compared with Focus in the same study populations, although performance also varied across studies. These results contrast with studies using North American samples, which have found specificity of Focus of 96% to 97% in antenatal and STI clinic populations.³¹ The study of Nigerian samples included in our review also evaluated samples from 6 countries in Asia and South/Central America, and found specificity of 91% to 95% in these non-African settings.²² In that study, sensitivity was also very high (98%–100%) in all but 1 site (91% in Songkla, Thailand).

Possible causes of discrepant results between Focus, Kalon, and UW-WB may include a low sensitivity of UW-WB and Kalon to detect either recent HSV-2 seroconversions,³² presence of African HSV-2 strain variants not detectable by the commercial tests, or selective cross-reactivity of antibodies to non-HSV-2 proteins in the Focus test. A study in Seattle found poor sensitivity of UW-WB to detect early HSV-2 seroconversions.³³ In this study, Focus detected seroconversion earlier (21 days) than UW-WB (68 days) or Kalon (120 days).³³ Similarly, in a study comparing performance of Focus and Kalon directly among genital ulcer disease patients in Ghana and the Central African Republic, Focus detected significantly more HSV-2 seroconverters among patients with proven first episode genital HSV-2 at Day 14 following presentation (77% vs. 23%; P = 0.01).³⁴ Authors of 2 studies have proposed the use of an HSV-2 recombinant gG ELISA inhibition assay as an alternative gold standard to the WB.^25,29 This assay measures antibody binding to multiple epitopes of HSV-2 gG2 present in cell culture lysates to inhibit the binding of gG2-specific antibodies to recombinant gG2.^25,29 However, the assay has not been evaluated exhaustively and will require further study. Sensitivity may also be lower of Kalon and UW-WB then Focus, as a result of cross-reactivity with HSV-1 antibodies is also a possibility. HSV-1 in many African settings is more than 90% compared with 50% to 70% in North America.²¹

The lower specificity of Focus observed in these African populations may be in part because of cross-reactivity with unidentified antibodies that might be common in African populations but are uncommon elsewhere. It may also be due to nucleotide polymorphism in the gG2 sequence that are specific to African populations. Although a European study³⁵ found that the gG2 epitope was highly conserved, HSV-2 sequences in African populations have not yet been characterized in the GenBank database.³⁶ Sequencing of HSV-2 strains in African populations may also help explain some of the geographical differences in test performance. HIV infection appears to cause a reduction in specificity. The reasons for this are not clear, although it is plausible that immune responses associated with HIV infection, such as immunoglobulin G polyclonal immune stimulation, may cause cross-reactions in the test.³⁷ Finally, operator error may cause low specificity, for example whether there is contamination of samples by splashing during dilution of sera or during transfer of diluted sera to test plates. Ashley-Morrow et al. showed that as little as 0.5 μl of added positive serum could convert a Focus negative to a low positive.²² It should be noted that this low specificity of Focus was not observed in all African populations. Two studies among HIV negative participants in Zimbabwe did not find lower specificity.

Biokit rapid assay results were highly variable across 2 studies. This could be due to inherent differences in the populations studied or, more likely, to variability in reading criteria for designating a test as positive.³⁰

The studies reviewed show that in these sub-Saharan African populations a useful strategy to improve performance is to use an index cut-off higher than that recommended by the manufacturer. Optimal performance was obtained when the cut-off was increased to 1.5 for Kalon, and to 3.4 or 3.5 for Focus. Studies have evaluated algorithms using Focus as the initial screening test, at either manufacturer's or higher cut-off OD index values, followed by confirmatory testing of all positives with Kalon, but these strategies have not proven to be advantageous over using Kalon alone^23,29 with some exceptions.³⁰

Our review has some limitations. Heterogeneity was observed in most analyses, implying that the performance of the tests may have differed by study population. The summary estimates should therefore be interpreted with caution. Also, the studies that were included in the meta-analysis evaluating the effect of increasing the cut-off OD did not always use the same cut-off OD. This may limit the comparability of these studies. However, in 5 of 7 studies, the assays were optimized for the study population and the most appropriate higher cut-off OD was used, thereby maximizing performance of the tests for these populations.

To summarize, the specificity of commercially available HSV-2 serological tests using African samples appear generally inferior to that reported from industrialized countries. There is a large variation in performance depending on geographical location and characteristics of the study populations. Therefore, studies using HSV-2 testing would benefit from an evaluation of test performance in the proposed study population, bearing in mind the aims of testing, for example, estimation of prevalence, establishing etiology of genital ulcers, estimating the effect of HSV-2 on risk of HIV acquisition, or infectivity. Different assays could be used for different purposes, either requiring high sensitivity or high specificity. Increasing the cut-off OD to 1.5 for Kalon and between 2.2 and 3.5 may improve test performance. An assay that is well suited for African populations is needed.