Sexually Transmitted Diseases:
Reliability and Criterion-Related Validity of Self-Report of Syphilis
Fisher, Dennis G. PhD*; Reynolds, Grace L. DPA*; Creekmur, Beth BA*; Johnson, Mark E. PhD†; Deaugustine, Netti BA‡
From the *Center for Behavioral Research and Services, California State University, Long Beach, California; †Behavioral Health Research and Services, University of Alaska, Anchorage, Alaska; and the ‡Department of Health and Human Services, City of Long Beach, Long Beach, California
This research was supported in part by contract number H-700939-1 from the County of Los Angeles, Office of AIDS Programs and Policy. Further support was provided in part by contract 28569 from the City of Long Beach, California, Department of Health.
Correspondence: Dennis G. Fisher, PhD, Center for Behavioral Research and Services, 1090 Atlantic Avenue, Long Beach, CA 90813. E-mail: email@example.com.
Received for publication March 11, 2006, and accepted August 14, 2006.
Objective: To determine the test-retest reliability, sensitivity and specificity, and criterion-related validity of the Risk Behavior Assessment (RBA) syphilis questions. The RBA is a standardized instrument that has been used in several studies of STDs in drug users.
Methods: For the test-retest reliability study, 219 injection drug users completed the RBA twice within a 48-hour period. To determine criterion-related validity, 207 individuals, who also completed the RBA, were tested with the rapid plasma reagin test (RPR), and 206 individuals were also tested with the Serodia Treponema pallidum particle agglutination test (TP-PA).
Results: The test-retest reliability for the question “How many times have you been told by a doctor or a nurse that you had syphilis?” was 0.78. The test-retest reliability for the question “In what year were you last treated for syphilis?” was 0.89. For the comparison of self-report with the RPR test, the sensitivity of self-report was 46.2% and the specificity was 95.7%. For the comparison of self-report with the TP-PA test, the sensitivity of self-report was 37% and the specificity was 97.7%.
Conclusions: Self-reports of syphilis infection history were found to have good reliability, excellent specificity, and moderate sensitivity. These characteristics need to be taken into account in any study using these self-report items.
AS A RESULT OF THE HIGH costs of laboratory tests, self-report is often the only measure used in research regarding sexually transmitted diseases (STDs). Using self-report of disease status without considering the data’s reliability and validity can seriously compromise the value of any research findings. Therefore, reliability and validity of disease self-report status is important. This article reports on the reliability and criterion-related validity of self-report of syphilis. The syphilis questions are from the Risk Behavior Assessment (RBA; National Institute on Drug Abuse),1 a standardized instrument used in several studies of STDs in drug users.2–7 The RBA measures sexual and drug-risk behaviors, demographics, and history of previous diagnosis with selected infectious diseases, including syphilis. Most of the items have good test–retest reliability and recent drug use has good validity.8–10
In 2 separate studies, we determined the test–retest reliability,3,11,12 sensitivity and specificity, and criterion-related validity of the RBA syphilis items. Reliability or stability of a measure is an essential precursor in describing properties of a measure. Sensitivity and specificity can be used to express criterion-related validity when the criterion (or gold standard) takes only 2 values such as positive versus negative laboratory test results.13 In our study, we compared self-report data and biomarkers as revealed by both the ASI Rapid Plasma Reagin Card Test14 and the Serodia TP-PA qualitative gelatin particle agglutination assay.15–17 Once a participant has had syphilis, the Serodia assay (TP-PA) will remain reactive for life.15
In study 1, to determine test–retest reliability, we recruited out-of-treatment injection drug users from the national NIDA Cooperative Agreement (CA). The data collection took place during March 1992. The 11 sites participating were Anchorage, Alaska; Denver, Colorado; Detroit, Michigan; Houston, Texas; Long Beach, California; Miami, Florida; New York, NY; Philadelphia, Pennsylvania; Portland, Oregon; San Francisco, California; and Tucson, Arizona. The sample consisted of 219 participants. Eligibility for inclusion was being at least 18 years old, not intoxicated or mentally impaired, self-reported injection drug or crack cocaine use within the last 30 days, visible signs of injection drug use (i.e., track marks), and/or positive ONTRAK18,19 urinalysis confirming morphine or cocaine metabolites and no enrollment in a drug treatment program within the last 30 days. Table 1 provides participants’ demographic information for study 1. These participants completed the RBA twice within a 48-hour period. The 48-hour time period was originally chosen for the comparison of self-report of recent drug use with urine test data reported elsewhere.8,9 The specific questions analyzed in this study are: “How many times have you been told by a doctor or a nurse that you had syphilis?” and “In what year were you last treated?”
In study 2, to determine criterion-related validity, we analyzed data from participants tested for syphilis within Service Planning Areas 5 (West) and 8 (South Bay) of Los Angeles County. Data for this study were collected from March 15, 2005, through April 5, 2006. To be eligible for this study, individuals fell into one of the following Los Angeles County-defined categories: men who have sex with men (MSM), men who have sex with men and women (MSMW), men who have sex with men/injection drug users (MSM/IDUs), heterosexual injection drug users (HIDUs), female injection drug users (FIDUs), women at sexual risk (WSR), and transgenders at sexual risk/transgender/injection drug user (TSR/T/IDUs). The only heterosexuals considered to be at risk under this system are the HIDU, FIDU, and the WSR. Table 2 provides participants’ demographic information for study 2. Most participants were male. The major race groups were white, black, and Hispanic. Most of the sample was categorized into either the MSM or the MSMW behavioral risk group. Mean age was 38.59 years (standard deviation = 11.0 years).
For test–retest reliability, in response to the question “How many times have you been told by a doctor or a nurse that you had syphilis (syph)?” the 48-hour test–retest reliability as measured by a Spearman correlation was 0.78 (95% confidence interval [CI] = 0.72–0.82). In response to the question “In what year were you last treated for syphilis?” the test–retest reliability (Pearson) was 0.89 (95% CI = 0.68–0.96). Based on the standard set by Nunnally that the minimum reliability for research used to compare groups is 0.70, both of the RBA syphilis measures had adequate reliability.20
For criterion-related validity, of the participants, 207 were tested with the rapid plasma reagin (RPR) test; of these, 14 were positive for a proportion positive of 6.76% (95% CI = 3.75–10.1). For the TP-PA test, 206 were tested and 27 were positive for a proportion positive of 13.11% (95% CI = 8.82–18.89). All those testing positive with RPR also tested positive with the TP-PA. For the sensitivity and specificity analyses, the question “How many times have you been told by a doctor or a nurse that you had syphilis (syph)?” was recoded into an ever/never format. Using the RPR test as the criterion, the sensitivity of self-report data was 46.2% and the specificity was 95.7%. The positive predictive value was 43% (95% CI = 18–71) and the negative predictive value was 96% (95% CI = 92–98). Using the TP-PA as the criterion, the sensitivity of self-report was 37% and the specificity was 97.7%. The positive predictive value was 71% (95% CI = 42–92) and the negative predictive value was 91% (95% CI = 86–95). Combining a negative RPR with a positive TP-PA, which would presumably give those people who had syphilis and were treated for it, gives a sensitivity of 28.6% and specificity of 94.6%. The positive predictive value was 29% (95% CI = 8–58) and the negative predictive value was 95% (95% CI = 90–97). Even acknowledging that neither of these 2 laboratory tests are a perfect gold standard, these results are comparable to or exceed those reported for other self-reported infectious diseases on the RBA. For example, for hepatitis, specificity ranges from 79% to 98% and sensitivity ranges from 14% to 38%.3,5 For HIV, specificity is 99% and sensitivity is 32%.21
At least 2 possible reasons exist for why the sensitivity values were not higher. First, it may be that our study discovered cases of syphilis that had not previously been diagnosed. Second, it is possible that the respondent did not understand what the doctor or nurse was telling them and thought that they were answering our question correctly when they were not. Similarly, there are at least 2 possible explanations for why sensitivity is higher on syphilis compared with hepatitis self-report. It may be that individuals with acute hepatitis are less likely to seek medical assistance for their symptoms and thus less likely to have the hepatitis diagnosed. Injection drug users (IDUs) are commonly sick from one cause or another. It must also be noted that psychometric estimates for drug users do not necessarily transfer to other groups of people. Furthermore, IDUs may confuse withdrawal symptoms with hepatitis symptoms. When we informed participants who reported not having been told that they had hepatitis that they were infected, the participants usually acknowledged being very sick once and surmised that was the time they had been infected with hepatitis but had not sought medical assistance. Possible difference in underlying base rates of incidence and prevalence is also a confounding factor. Because the syphilis prevalence is low, the specificity is expected to be high. In addition, a population of men who have sex with men is very different from a population of drug injectors when it comes to sexually transmitted infections. Symptoms for syphilis in men compared with symptoms of hepatitis are significantly different. Some differences in this study could be attributed to the nature of the infection (manifesting itself on the sex organs) of a sexually active population (MSM) who may be better educated and have better access to health care compared with the nature of hepatitis infection (flu-like symptoms, general malaise) predominantly found in an IDU population with lower education and less access to health care.
In conclusion, self-reports of syphilis infection history by injection drug users were found to have good reliability, excellent specificity, and moderate sensitivity. However, it is important to note that the use of both self-report and confirmatory laboratory testing in studies to assess reliability and validity of sexually transmitted infections needs to be considered on a case-by-case basis because the measurement issues involved in each STD may be different, depending on the disease and the population being studied.
1. National Institute on Drug Abuse. Risk behavior assessment. In: U.S. Department of Health and Human Services. Vol ICPSR 3023: Inter-university Consortium for Political and Social Research, 2003.
2. Booth RE, Watters JK, Chitwood DD. HIV risk-related sex behaviors among injection drug users, crack smokers, and injection drug users who smoke crack. Am J Public Health 1993; 83:1144–1148.
3. Fisher DG, Kuhrt-Hunstiger TI, Orr SM, Davis DC. Hepatitis B validity of drug users’ self-report. Psychol Addict Behav 1999; 13:33–38.
4. Paschane DM, Fisher DG, Cagle HH, Fenaughty AM. Gonorrhea among drug users: An Alaskan versus a national sample. Am J Drug Alcohol Abuse 1998; 24:285–297.
5. Schlicting EG, Johnson ME, Brems C, Wells RA, Fisher DG, Reynolds GL. Validity of injecting drug users’ self report of hepatitis A, B, and C. Clinical Laboratory Science 2003; 16:99–106.
6. Siegal HA, Falck RS, Wang J, Carlson RG. History of sexually transmitted diseases infection, drug-sex behaviors, and the use of condoms among Midwestern users of injection drugs and crack cocaine. Sex Transm Dis 1995; 23:277–282.
7. Williams M, Elwood WM, Weatherby N, et al. An assessment of the risks of syphilis and HIV infection among a sample of not-in-treatment drug users in Houston, Texas. AIDS Care 1996; 8:671–682.
8. Dowling-Guyer S, Johnson ME, Fisher DG, et al. Reliability of drug users’ self-reported HIV risk behaviors and validity of self-reported recent drug use. Assessment 1994; 1:383–392.
9. Needle R, Fisher DG, Weatherby N, et al. Reliability of self-reported HIV risk behaviors of drug users. Psychol Addict Behav 1995; 9:242–250.
10. Fisher DG, Needle R, Weatherby N, Brown B, Booth RE, Williams M. Reliability of drug user self-report. Presented at the IXth International Conference on AIDS, 1993, Berlin, Germany.
11. Van Duynhoven Y, Nagelkerke N, Van De Laar M. Reliability of self-reported sexual histories: Test–retest and interpartner comparison in a sexually transmitted diseases clinic. Sex Transm Dis 1999; 26:33–42.
12. Sneed CD, Chin D, Rotheram-Borus MJ, et al. Test–retest reliability for self-reports of sexual behavior among Thai and Korean respondents. AIDS Educ Prev 2001; 13:302–310.
13. Myer L, Morroni C, Link BG. Impact of measurement error in the study of sexually transmitted infections. Sex Transm Infect 2005; 80:318–321.
14. ASI RPR Screening Test for Syphilis. Springville, UT: Arlington Scientific, Inc, 1997.
15. Serodia-TP-PA: Reagents for the Detection of Antibodies to Treponema pallidum.
Malvern, PA: Fujirebio Diagnostics, Inc, 000–975 09/02.
16. Aktas G, Young H, Moyes A, Badur S. Evaluation of the Serodia Treponema pallidum
particle agglutination, the Murex Syphilis ICE and the Enzywell TP tests for serodiagnosis of syphilis. Int J STD AIDS 2005; 16:294–298.
17. Pope V, Fears MB, Morrill WE, Castro A, Kikkert SE. Comparison of the Serodia Treponema pallidum
particle agglutination, Captia Syphilis-G, and SpiroTek Reagin II tests with standard test techniques for diagnosis of syphilis. J Clin Microbiol 2000; 38:2543–2545.
18. OnTrak TesTstik COC.
Lake Forest, CA: Varian, Inc, 2003:0766518.
19. OnTrak TesTstik MOR. Lake Forest, CA: Varian, Inc, 2003:0766534.
20. Nunnally JC, Bernstein IH. Psychometric Theory, 3rd ed. New York: McGraw-Hill, Inc, 1994.
21. Fisher DG, Reynolds GL, Jaffe A, Johnson ME. Reliability, sensitivity, and specificity of self-report of HIV test results. AIDS Care. In press.
© Copyright 2007 American Sexually Transmitted Diseases Association