Many sexually transmitted disease (STD) clinics have implemented express care models, which include a testing-only visit option without a clinician evaluation, to meet the high demand for services in the context of low resources1 and increasing rates of many STDs.2 Express care may increase clinic efficiency and reduce long-term costs, allowing limited clinician time and resources to be focused on patients in need of same-day treatment and facilitating rapid access to STD screening.3 However, there is also the potential to miss health conditions of public health importance that could have been identified during a clinician evaluation.4
Sexually transmitted disease clinics have implemented a variety of express care models, each with its own method of patient history collection, triage process, and distinct set of criteria for determining patients' visit types (Table 1).5–13 Triage criteria may be disease focused (e.g., patient has genital symptoms) or health service focused (e.g., female patient who wants to discuss contraception with a clinician), depending on the patient population and local public health priorities and resources. Although not reported in the literature, we have observed that some STD clinics use a simple triage algorithm including only whether the patient has any symptoms or had known contact to a sex partner with an STD or HIV (hereafter referred to as “contact to STD”). To our knowledge, only one study has attempted to formally construct triage algorithms that reliably predict who will receive specific STD diagnoses that could be missed in testing-only visits.14 Using medical record data from 3 STD clinics, Xu and colleagues14 found that age and/or specific STD-related symptoms were associated with each of 4 types of diagnoses that could be missed in express visits: genital herpes or warts, trichomoniasis, symptomatic vulvovaginal candidiasis or bacterial vaginosis, and nongonococcal nonchlamydial urethritis. However, age and/or specific symptoms had low discriminatory ability for predicting those diagnoses, even when used in combination. Other studies have evaluated clinics' respective express care models in different ways, but no published studies to date have compared the relative discriminatory ability of multiple triage algorithms for predicting who needs to see a clinician within the same population.
We recently reported an evaluation of the express care triage algorithm currently used in the Public Health – Seattle & King County (PHSKC) STD Clinic in Seattle, Washington.12 This algorithm accurately identifies patients with an infection or syndrome that can be diagnosed and treated at that visit, but it triages less than 25% of patients to express care which limits efficiency. The ideal algorithm would triage a substantial percentage of patients to express care without inappropriately triaging any patients to express care.
The goal of the present study was to construct and evaluate alternative triage algorithms to determine which approach most reliably identifies clinically relevant cases in our clinic while maximizing triage to express care. To achieve this goal, we (1) constructed and validated an optimized algorithm that maximizes correct triage of patients between express visits and standard visits with a clinician evaluation, and (2) compared the performance of the optimized algorithm, the PHSKC STD Clinic's current algorithm, and a “simple” algorithm that considers only presence of symptoms and contact to STD.
Study Design and Sample
The PHSKC STD Clinic implemented systematic computer-assisted self-interview (CASI) assessments for collecting patient history and developed a computerized algorithm to automate triage in 2010.15 However, because of the staffing model in place, almost all patients continued to receive a clinician evaluation irrespective of triage status.12 Therefore, the clinic database contains both CASI responses and outcomes of a clinician evaluation for each patient over a 5-year period.
This cross-sectional study included patients attending the PHSKC STD Clinic who completed a CASI at a new problem visit between October 1, 2010, and June 30, 2015. The 36% of new problem visits without a CASI were excluded.12 Our analysis excluded transgender persons because their patient history was collected in-person rather than through CASI during the analysis period. For individuals with multiple new problem visits with a CASI during this period, we randomly selected one visit for inclusion in the analysis to maintain independence of observations.
The outcome for this study, “need for a standard visit,” was defined using electronic medical record data. The outcome definition was identical to that used in our prior study reporting performance of the current triage algorithm.12 We coded each visit to indicate whether the patient needed a standard visit or could have received an express visit based on the clinician assessment. Our definition of needing a standard visit was disease focused and intended to identify patients with an infection of public health importance that could be diagnosed and treated at that visit. We classified patients as needing a standard visit if they reported key symptoms (anogenital symptoms, abdominal pain [for women], body rash, and symptoms concerning to the patient for acute HIV infection) to the CASI or clinician, received any empiric treatment, or were diagnosed as having a key infection or syndrome at the same visit. To ascertain symptoms concerning to the patient for acute HIV, our CASI asked: Did you come to the clinic to test for HIV because of symptoms you think may be a sign that you were recently infected with HIV? Key infections or syndromes were possible to diagnose with clinical examination and point-of-care testing alone and of public health importance due to the potential for ongoing transmission or health consequences of delayed treatment. These included primary or secondary syphilis; urethral, cervical, or rectal Neisseria gonorrhoeae detected on Gram stain; nongonococcal urethritis; epididymitis; proctitis; mucopurulent cervicitis; pelvic inflammatory disease; bacterial vaginosis; vaginal candidiasis; trichomoniasis; urinary tract infection; genital ulcer of unknown etiology; skin and soft tissue infection (i.e., cellulitis); and herpes simplex virus. Patients who did not meet any of these criteria were classified as eligible for express care. Importantly, HIV diagnosed by point-of-care test was not a criterion for a standard visit because a patient with a positive HIV test result during an express visit can be routed to see a clinician at that time. Patients with symptomatic acute HIV were defined as needing standard visits due to their symptoms.
To develop the optimized algorithm, we selected 11 potential predictors of need for a standard visit. All are self-reported measures collected by CASI that have been highly associated with the key infections or syndromes that we determined would require a standard visit. These included age (in years),2,14,16,17 key symptoms,5,14,18 sore throat, positive STD result from another provider and needs treatment, contact to STD, sex partner with symptoms,18 same-sex partner in the past year,2,17,18 number of sex partners in the past 2 months,17,18 popper (for men who have sex with men only)19 or methamphetamine use in the past year,20,21 crack cocaine use in the past year, and ever had transactional sex.22
After optimizing the triage algorithm, we compared its performance to the PHSKC STD Clinic's current algorithm and a simple algorithm. The current algorithm triages patients to a standard visit for 13 disease- and health service-focused reasons (Table 1). The simple algorithm triages patients to a standard visit for 2 disease-focused reasons: report of any symptoms (including sore throat) or contact to STD.
We classified patients who were mistriaged by an algorithm into 2 categories. We defined patients who needed a standard visit but were triaged to express visits by the algorithm as “underserved” and patients who did not need a standard visit based on the study definition but were triaged to standard visits by the algorithm as “overserved.” This terminology reflects the disease-focused outcome for this analysis.
Construction and Validation of the Optimized Algorithm
We used independent development and validation samples to construct and validate the optimized algorithm. Separately for men and women, the new problem visits was randomly divided in half, resulting in sex-specific development and validation samples (Supplement, http://links.lww.com/OLQ/A266). We assumed that all patients with key symptoms should be evaluated by a clinician. We did not make this assumption for contact to STD because, in practice, we have noticed variability in what patients report as contact to STD, some of which do not require evaluation (e.g., contact from >1 year ago, contact for which the patient already received treatment, and contact to an STD for which treatment is not indicated). Therefore, we created sex-specific development and validation “subsamples” that excluded individuals with key symptoms, which were used to develop and validate the initial sex-specific optimized algorithms.
We subjected each development subsample to classification and regression tree (CART) analysis, considering the 11 potential predictors. CART analysis is a nonparametric, empiric statistical method for developing prediction models.23 It yields a sequence of nested risk stratification trees (i.e., decision trees) composed of progressive binary splits that divide patients into increasingly homogenous groups in terms of their outcome value. At each split, CART considers each predictor and uses that which best separates patients with and without the outcome. For continuous predictors, CART considers the cutoff value that best predicts the outcome. Predictors may be used more than once within the tree or not at all. We used CART analysis because it does not require assumptions about the underlying distributions of the predictors and is well suited to considering complex interactions between predictors that are difficult to model with logistic regression.
We considered the optimal tree to be that with the lowest misclassification error rate (mistriaged visits/total visits) when the trees were applied to the validation subsample. Subsequently, we programmed the final optimized algorithm into Stata 13 (StataCorp, College Station, TX), including the decision rules contained in each of the sex-specific optimal trees and the key symptoms criterion. This final optimized algorithm reflects what would be used in clinic operations.
Comparison of Triage Scenarios
We compared 3 triage scenarios in the full validation samples for men and women, each considering a separate triage algorithm: the final optimized algorithm, the PHSKC STD Clinic's current algorithm, and the simple algorithm. For each scenario, we estimated the sensitivity, specificity, and area under the receiver operating curve (AUC) for appropriately triaging patients between express visits and standard visits. The criterion standard was need for a standard visit based on the clinician evaluation. We also calculated the percent express visits to approximate the efficiency of each algorithm. Finally, we characterized the patients mistriaged by the optimized and simple algorithms. The characteristics of patients mistriaged by the current algorithm were published previously.12
This study was approved by the University of Washington Human Subjects Division. Analyses were conducted in Salford Predictive Modeler 7 (Salford Systems, San Diego, CA) and Stata 13.
Between October 1, 2010, and June 30, 2015, 18,653 patients completed a CASI at 32,102 new problem visits. After randomly selecting 1 visit for patients with multiple visits (n = 13,449 visits excluded), there were 18,653 unique patients with a CASI, including 13,968 men (75%) and 4685 women (25%; Table 2). Less than half of men (45%) reported sex with men in the past year. Most patients were seeking care for key symptoms (49% of men, 60% of women).
For men, the full development and validation samples each contained 6984 patients; the development and validation subsamples, which excluded men with key symptoms, contained 3443 and 3344 patients, respectively. For women, the full development and validation samples contained 2343 and 2342 patients, respectively; the development and validation subsamples contained 873 and 857 patients, respectively. The optimal risk stratification trees for predicting need for a standard visit among men and women without key symptoms are presented in Figure 1. Among men, the optimal tree included 3 predictors: contact to STD, positive STD result from another provider, and sex partner with symptoms. Among women, the optimal tree included 5 predictors: contact to STD, positive STD result from another provider, sex partner with symptoms, age, and number of sex partners in past 2 months. The optimal tree for women included interactions among some predictors, so not all women meeting an individual criterion would be triaged to standard visits. The criteria in the final optimized algorithm, current PHSKC STD Clinic algorithm, and simple algorithm are summarized in Table 3.
When we considered the 3 triage scenarios in the full validation samples for men and women, the optimized, current, and simple algorithms appropriately triaged 8418 (90%), 7943 (85%), and 8273 (89%) patients, respectively (Table 4). The optimized and simple algorithms had similar overall performance (AUC) for men (optimized, 0.88; simple, 0.87) and women (optimized, 0.90; simple, 0.89) (Table 5). The AUC for the current algorithm was lower (men, 0.83, women, 0.65) due to lower specificity. However, the current algorithm had the highest sensitivity for men (95%) and women (98%). The optimized and simple algorithms triaged a similar percentage of patients to express visits (31% and 33%, respectively), indicating similar efficiency. The current algorithm triaged a lower percentage of patients to express visits (23%), primarily due to the health service–focused criteria for needing a standard visit.
The primary reasons that men who would have been underserved by the optimized and simple algorithms needed a standard visit were report of key symptoms to the clinician that were not disclosed to the CASI (47% and 34% of underserved men, respectively) and receipt of empiric therapy for something other than contact to STD despite being asymptomatic (27% and 26%, respectively). The primary reason that women who would have been underserved needed a standard visit was diagnosis with a key infection or syndrome despite being asymptomatic (45% for both algorithms). Compared with the current algorithm, use of the simple algorithm during this 5-year period would have delayed diagnosis and treatment of 3 cases of early syphilis and 55 cases of gonorrhea. For all 3 early syphilis cases and 49 gonorrhea cases (89%), the patient reported having a positive STD result from another provider.
The primary reason that men and women would have been overserved by the optimized and simple algorithms was report of contact to STD to the CASI when they did not subsequently receive empiric therapy during the clinician evaluation (70%–84% of overserved patients). In addition, a substantial percentage of overserved men (16%) and overserved women (27%) were triaged to standard visits by the optimized algorithm because they reported having a sex partner with symptoms to the CASI but did not subsequently receive empiric therapy or a key diagnosis during the visit.
We found that the performance of a simple 2-component algorithm for triaging STD clinic patients was comparable to a fully optimized algorithm developed with CART. The current and more complex PHSKC algorithm had slightly higher sensitivity for the disease-focused outcome but triaged substantially fewer patients to express visits than each of the other algorithms. In addition to the factors in the simple algorithm (symptoms and contact to STD), the optimized algorithm included having a positive STD result from another provider, having a sex partner with symptoms, and age and number of sex partners in the past 2 months (for women). In most settings, the benefit of having a simple triage algorithm would outweigh the minute gain in sensitivity with the optimized algorithm.
Our finding that STD-related symptoms and contact to STD are the most useful factors for excluding patients from express care provides evidence to support a common clinical practice. It is generally consistent with other studies that have evaluated the predictive ability of various criteria for identifying specific STD diagnoses14 and STD test positivity.5 Our CART analysis to develop an optimized algorithm did not indicate much benefit from adding other potential predictors, providing confidence that the simple algorithm is appropriate for identifying patients in need of same-day treatment.
However, the appropriate triage criteria for a clinic will depend on the patient population, local public health priorities and resources, motivation for implementing an express care model, and relative value clinic leadership place on resources saved and other benefits associated with more extensive use of express care versus the cost of missed opportunities to diagnose and treat patients when they initially present for care. For example, in our clinic, although use of the simple algorithm during the analysis period would have increased clinic efficiency and access to STD screening, treatment of some cases of early syphilis (<1 case per year) and gonorrhea (~12 cases per year) would have been delayed. We would need to add a third criterion, positive STD result from another provider, to the simple algorithm to identify most of these cases, although this issue may not be applicable in some settings. Moreover, our study outcome focused solely on infections and syndromes that could be detected and treated at that visit. This is a central purpose of STD clinics, and in the context of limited resources, triaging patients based on key symptoms and recent contact to STD may be sufficient to identify patients needing a clinician evaluation while facilitating rapid STD screening for many patients (>33%) without clinician assessment. However, the mission of many STD clinics includes providing health services to vulnerable populations. In this context, an algorithm that includes providing clinician visits to patients needing contraceptive services, linkage to HIV and hepatitis C treatment, or other priorities may be more appropriate. Notably, during most of the study period, the PHSKC STD Clinic was not yet systematically identifying candidates for HIV preexposure prophylaxis.24 This is now a priority and requires ascertainment of risk factors different from those of clinical triage.
This study had important limitations. First, our analysis was restricted to visits with a CASI at a single STD clinic that serves a patient population which is currently and increasingly composed of predominantly men who have sex with men. Hispanic patients were less likely to complete a CASI than non-Hispanic patients.12 Although we used independent development and validation samples to construct the optimized algorithm, our findings may not be generalizable to other patient populations. Second, there was likely some misclassification of our outcome variable if some patients truly received express visits. Finally, our disease-focused outcome did not incorporate all aspects of high-quality STD services (e.g., patient preferences).
In conclusion, we evaluated 3 triage algorithms that can be implemented in STD clinics to achieve goals of improved clinic efficiency or improved disease detection. We found that simple triage criteria of symptoms and contact to STD can have high sensitivity for identifying patients with infections or syndromes that can be diagnosed at the visit, while also maximizing clinic efficiency. Additional health service–focused criteria can be layered on top of these simple criteria, depending on local resources and public health priorities, while maintaining moderately high efficiency.
1. Leichliter JS, Heyer K, Peterman TA, et al. US public sexually transmitted disease clinical services in an era of declining public health funding: 2013–14. Sex Transm Dis 2017; 44:505–509.
2. US Centers for Disease Control and Prevention. Sexually Transmitted Disease Surveillance 2016. Atlanta, GA: Department of Health and Human Services, 2017.
3. Dombrowski JC, Golden MR. Modernizing operations to improve efficiency and refine the role and mission of sexually transmitted infection clinics. Sex Transm Dis 2013; 40:81–84.
4. Rietmeijer CA. The pros and cons of the express visit option. Sex Transm Dis 2013; 40:62–63.
5. Heijman TL, Van der Bij AK, De Vries HJ, et al. Effectiveness of a risk-based visitor-prioritizing system at a sexually transmitted infection outpatient clinic. Sex Transm Dis 2007; 34:508–512.
6. Wong W, Broad J, Rutledge T, et al. Impact of a Pilot Fast-Track Screening Program for a High-Volume STD Clinic—Chicago, Illinois. Presented at: 2006 STD Prevention Conference; 2006; Jacksonville.
7. Shamos SJ, Mettenbrink CJ, Subiadur JA, et al. Evaluation of a testing-only “express” visit option to enhance efficiency in a busy STI clinic. Sex Transm Dis 2008; 35:336–340.
8. Gratrix J, Bergman J, Brandley J, et al. Impact of introducing triage criteria for express testing at a Canadian sexually transmitted infection clinic. Sex Transm Dis 2015; 42:660–663.
9. Rukh S, Khurana R, Mickey T, et al. Chlamydia and gonorrhea diagnosis, treatment, personnel cost savings, and service delivery improvements after the implementation of express sexually transmitted disease testing in Maricopa County, Arizona. Sex Transm Dis 2014; 41:74–78.
10. Yeung A, Bush M, Cummings R, et al. Use of computerized medical records to determine the feasibility of testing for chlamydia without patients seeing a practitioner. Int J STD AIDS 2010; 21:755–757.
11. Borrelli JM, Paneth-Pollak R, Wright S, et al. The Impact of Introducing “Express Visits” for Asymptomatic Persons Seeking STD Services in a Busy Urban STD Clinic System, New York City, 2005–2006. Presented at: 2008 STD Prevention Conference [C2d]; 2008; Atlanta.
12. Chambers LC, Manhart LE, Katz DA, et al. Evaluation of an automated express care triage model to identify clinically relevant cases in a sexually transmitted disease clinic. Sex Transm Dis 2017; 44:571–576.
13. Knight V, Ryder N, Guy R, et al. New Xpress sexually transmissible infection screening clinic improves patient journey and clinic capacity at a large sexual health clinic. Sex Transm Dis 2013; 40:75–80.
14. Xu F, Stoner BP, Taylor SN, et al. “Testing-only” visits: an assessment of missed diagnoses in clients attending sexually transmitted disease clinics. Sex Transm Dis 2013; 40:64–69.
15. Dombrowski JC, Kerani RP, Golden MR. STD Clinic Triage Based on Computer-Assisted Self Interview: The King County Experience. Quebec: Presented at: 19th meeting of the International Society for Sexually Transmitted Diseases Research [O5-S1.06], 2011.
16. Satterwhite CL, Torrone E, Meites E, et al. Sexually transmitted infections among US women and men: Prevalence and incidence estimates, 2008. Sex Transm Dis 2013; 40:187–193.
17. Hughes G, Brady AR, Catchpole MA, et al. Characteristics of those who repeatedly acquire sexually transmitted infections: A retrospective cohort study of attendees at three urban sexually transmitted disease clinics in England. Sex Transm Dis 2001; 28:379–386.
18. US Centers for Disease Control and Prevention. In: Sexually Transmitted Diseases Treatment Guidelines 2015. Atlanta, GA: Department of Health and Human Services, 2015.
19. Tabet SR, Krone MR, Paradise MA, et al. Incidence of HIV and sexually transmitted diseases (STD) in a cohort of HIV-negative men who have sex with men (MSM). AIDS 1998; 12:2041–2048.
20. Newman LM, Dowell D, Bernstein K, et al. A tale of two gonorrhea epidemics: results from the STD surveillance network. Public Health Rep 2012; 127:282–292.
21. Chew Ng RA, Samuel MC, Lo T, et al. Sex, drugs (methamphetamines), and the Internet: Increasing syphilis among men who have sex with men in California, 2004–2008. Am J Public Health 2013; 103:1450–1456.
22. Marx R, Aral SO, Rolfs RT, et al. Crack, sex, and STD. Sex Transm Dis 1991; 18:92–101.
23. Strobl C, Malley J, Tutz G. An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychol Methods 2009; 14:323–348.
24. Golden MR, Lindquist S, Dombrowski JC. Public Health–Seattle & King County and Washington State Department of Health Preexposure Prophylaxis Implementation Guidelines, 2015. Sex Transm Dis 2016; 43:264–265.
Supplemental Digital Content
© Copyright 2018 American Sexually Transmitted Diseases Association