At the time of writing of this article, the United States was in the process of distribution of at-home antigen tests to every US resident via the United States Postal Service as an added measure for reducing further spread of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infections.1 The antigen tests are considered to be the gold standard for measuring infectiousness and thus a primary diagnostic tool for clinicians in prescribing quarantine time or management per infection protocols. These tests have a high specificity (∼90% or higher) and a low sensitivity (∼70% or lower), with lower sensitivity reported for asymptomatic individuals.2–6 To improve the sensitivity of the antigen tests, manufacturers require patients to repeat the test, to rely on the clinical review of their symptoms, or to get a confirmatory reverse transcription polymerase chain reaction (RT-PCR) test, which is considered the gold standard for diagnosing viral infection.7,8
Clinical diagnosis of coronavirus disease-2019 (COVID-19) from its symptoms is fraught with difficulty. Simple clinical rules for differential diagnosis of COVID-19 from respiratory diseases do not exist.9 Furthermore, COVID-19 is a systemic disease, with many different types of symptoms, and clinicians would need to rule out a variety of diseases including neurological and gastrointestinal diseases with similar symptoms.10 Finally, clinicians are often not available in the community where the at-home antigen tests are done and where clinical review is needed the most. Machine learning methods offer solutions to automate clinicians' review of patients' symptoms in the community. The current study adds to the literature by constructing and validating a model for predicting RT-PCR test results based on symptom screening, at-home antigen testing, and vaccination status. The study examined whether computer-aided symptom screening11 can enhance the sensitivity of at-home rapid tests and, hence, increase the accuracy of identifying active SARS-CoV-2 infections.
To date, few studies on the diagnostic accuracy of at-home COVID-19 tests are available, other than those conducted by their manufacturers.12 Independent studies are needed to confirm these studies. This study provides one such independent evaluation. In addition, the authorization for at-home tests was given during COVID-19 emergency and new approvals will likely be needed once the emergency period ends, and COVID-19 is recognized as an endemic disease. The current study provides additional data on effectiveness of one of the Food and Drug Administration (FDA) emergency use-approved, at-home, antigen tests for COVID-19.
IRB APPROVAL
The project was approved by Western Institutional Review Board, IRB (study 1309332). The data collection effort was approved to be carried out by Virginia Commonwealth University, with their IRB (HM20022035) deferring to Western IRB. The analysis of de-identified data was approved by George Mason University IRB (1743684-1).
METHODS
This study used primary data sources in which data were collected during 2 different periods. In phase 1 of the study, 483 subjects were recruited online, if they had received a diagnostic test for an active SARS-CoV-2 infection in the last 30 days. Twenty-two subjects were excluded from the final analysis because their test results were inconclusive or not available in time. Phase 1 data were collected prior to vaccination being made widely available, and when the alpha variant of the novel coronavirus was predominant in the United States. In phase 2 of the study, data were collected through employees and patients of Virginia Commonwealth University Health Center. A total of 599 subjects were recruited based on whether they were asymptomatic, had recent exposure to SARS-CoV-2, or were symptomatic. Thirty-seven percent of initially recruited subjects were excluded from the analysis because of noncompliance with the study protocol, completing RT-PCR and/or at-home tests, and due to multiple entries into the study. Among remaining 374 subjects who were recruited in phase 2, 87% were vaccinated. At the time of phase 2 data collection, delta was the predominant variant of SARS-CoV-2 in the United States.
Phase 1 data were used to develop a computerized symptom screening method, using a combination of ordinary, and least absolute shrinkage and selection operator (LASSO), logistic regressions to predict COVID-19 RT-PCR test results. Regression may be sensitive to small variations in the training sample. Numerous solutions have been proposed for selecting robust variables in regressions.13 We created a robust model in 2 steps. First, we repeated logistic LASSO regressions in different samples of phase 1 data; and selected only those nonzero variables present in 95% of the regressions. Second, we used ordinary logistic regression, to estimate the relationship between RT-PCR test results with independent variables previously selected through repeated LASSO regressions. These 2 steps allowed robust estimation of impact of (1) single, (2) pair, and (3) triplets of independent variables on RT-PCR.
Phase 2 data were used to examine the accuracy of predicting COVID-19 diagnosis with (1) computerized symptom screening; (2) at-home rapid antigen testing; (3) the combination of computerized symptom screening and at-home rapid antigen testing; and (4) the combination of symptom screening and vaccination status. The latter analysis was performed by randomly dividing phase 2 data 10 times into training (80%) and testing (20%) sets. The model was trained on the training set and validated on the testing set. Hierarchical, nested, logistic regression models were constructed in phase 2. McFadden pseudo-R2 was used as a measure of percentage of variation in RT-PCR test results explained by the various screening methods.
Study design
The computerized symptom screening tool was developed using phase 1 data as training data set. Phase 2 data were used to test the accuracy of at-home antigen tests and the computerized symptom screening method, separately and in conjunction with one another. The accuracy of the screening methods, at-home antigen tests and the computerized symptom screening, was tested by comparing those methods with self-reported results of RT-PCR test results by phase 2 study subjects. Logistic regression was used to assess the percentage of variation in RT-PCR test results explained by different screening methods. The dependent variable was the RT-PCR test result. The independent variables were a combination of screening methods. The percentage of variation in RT-PCR test results explained by different screening methods was based on McFadden pseudo-R2.14 Initially, single variables were included in the regression. To examine whether additional variables added information and increased the McFadden percentage of variation explained, new variables were added, and regression analysis repeated.15
Study samples
This study relied on 2 independent data samples collected at different times. The data from phase 1 of the study were used to train and estimate the parameters of the computerized symptom screening. Phase 1 of the study enrolled 483 subjects who were tested for an active SARS-CoV-2 infection with RT-PCR in the last 30 days. At the time of recruitment, no at-home antigen tests were readily available. RT-PCR tests were done at a variety of laboratories. Subjects self-reported their test results.
The data collected during phase 2 of the study were used to test the accuracy of computerized symptom screening, at-home antigen tests, and the combination of 2 types of tests. Phase 2 data were also used to examine the accuracy of predicting COVID-19 diagnosis based on individual symptoms and vaccination status. This analysis was possible because vaccines were more widely available during phase 2 of the study as compared with phase 1. Since at-home COVID tests are used for different reasons, 3 cohorts of patients were enrolled in phase 2 study: (1) those with known recent exposure to COVID-19; (2) those experiencing symptoms; and (3) asymptomatic individuals with no known exposure. Participants were recruited through Virginia Commonwealth University (VCU) and VCU-affiliated Health System. Participants were asked to (1) complete the online symptom survey; (2) self-administer an at-home antigen COVID-19 test; (3) repeat the at-home test after a specified period; and (4) complete an RT-PCR test at a specifically designated facility. Participants were instructed to use VCU's Medical Center to complete their RT-PCR tests. Subjects had to physically go to the laboratory facility to complete the test. Subjects were paid to participate in phase 2 of the study ($25 for the symptom survey, $25 for the first at-home test, $25 for the second at-home test, and $50 for RT-PCR test completion).
Testing for SARS-CoV-2 infection
After signing an electronic consent in phase 2 of the study, subjects were provided with 2 at-home tests to be completed within 24 hours, with no longer than 36 hours between the tests. The QuickVue COVID-19 test, available through over-the-counter purchase, was used as the at-home tests (QuickVue At-Home OTC COVID-19 Test, Emergency Use Authorization 210269) in the study. The QuickVue test is a self-administered (unobserved) and requires anterior nasal swab specimen. Study subjects were asked to complete an RT-PCR test for detection of SARS-CoV-2 infection within 7 days from enrollment in the study at the designated clinic, VCU Medical Center. All services, including at-home and RT-PCR tests, were free of cost to the subjects.
Inclusion criteria
Both phase 1 and 2 study participants were required to be adults, 18 years and older, with a decisional capacity to consent, able to read/write in English, and willing and able to complete study-related procedures.
Sampling procedure
In phase 1, blocked sample design was not used. In phase 2, 3 groups of subjects were recruited: symptomatic, asymptomatic but not exposed, and subjects with a recent known exposure to COVID-19. Data for both samples were weighted to reflect the age, gender, and race distribution of the US population.
Study attrition
Table 1 reports the number of subjects in the study data, in phases 1 and 2, and the attrition within each cohort. In phase 1 cross-sectional sample, 22 subjects had inconclusive test results or, at the time of responding to the survey, did not know the status of their test results. In the longitudinal phase 2 sample, 37.06% of subjects across the 3 cohorts were excluded from the analyses because of either not meeting eligibility requirements, not completing the at-home test, or failing to comply with the study protocol to complete the necessary RT-PCR test.
Table 1. -
Sample Size and Attrition
|
|
Phase 2 Sample May 2021 to October 2021 Delta Variant |
| Period Prevalent Strain Sampling Method |
Phase 1 Sample November 2020 to January 2021 Alpha Variant Tested in Last 30 d |
Exposed Cohort |
Symptomatic Cohort |
Asymptomatic Cohort |
Total |
| Consented online |
483 |
186 |
210 |
203 |
599 |
| Duplicate responses |
0 |
10 |
10 |
0 |
20 |
| Missing RT-PCR test |
22 |
77 |
62 |
43 |
182 |
| Not eligible |
0 |
0 |
1 |
0 |
1 |
| Missing home tests |
Not available |
0 |
21 |
1 |
22 |
| Available for analysis |
461 |
99 |
116 |
159 |
374 |
Abbreviation: RT-PCR, reverse transcription polymerase chain reaction.
RESULTS
Table 2 describes the demographic distribution of the 2 study samples. The phase 1 sample was used to train the model for the computerized symptom screening. Most of the study participants in that sample were female (69%), 25 to 34 years old (28%), and White (71%).
Table 2. -
Characteristics of Study Sample
| Variable |
Values |
Training Data (Phase 1) |
Validation Data (Phase 2) |
| Number of cases |
483 (100%) |
599 (100%) |
| COVID-19 test results |
Negative |
330 (68%) |
338 (60%) |
|
Positive |
131 (27%) |
56 (10%) |
|
Results pending |
15 (3%) |
... |
|
Inconclusive |
7 (1%) |
4 (1%) |
|
Missing |
... |
160 (29%) |
| Age |
18-24 y |
84 (17%) |
101 (18%) |
|
25-34 y |
210 (43%) |
156 (28%) |
|
35-44 y |
156 (32%) |
123 (22%) |
|
45-54 y |
20 (4%) |
89 (16%) |
|
55-84 y |
13 (3%) |
74 (13%) |
|
Missing |
... |
17 (3%) |
| Gender |
Female |
279 (58%) |
384 (69%) |
|
Male |
203 (42%) |
174 (31%) |
|
Other |
... |
2 (0%) |
| Ethnicity |
Hispanic Latino |
60 (12%) |
22 (4%) |
|
Non-Hispanic/Latino |
401 (83%) |
498 (93%) |
|
Unknown |
22 (5%) |
16 (3%) |
| Race |
Other |
18 (4%) |
11 (2%) |
|
Asian |
25 (5%) |
27 (5%) |
|
Black or African American |
60 (12%) |
120 (22%) |
|
White |
380 (79%) |
395 (71%) |
| Vaccinated |
0 (0%) |
316 (87%) |
Table 3 provides the estimated robust coefficients. These coefficients provide the magnitude and the direction of impact of various robust variables on positive RT-PCR test results.
Table 3. -
Factors That Predict RT-PCR Test Results on Symptoms and Selected Demographic Characteristics
a
| Variable |
Coefb
|
Variable |
Coefb
|
| Intercept |
−2.68 |
White, shortness of breath, wheezing |
0.45 |
| Female |
1.24 |
White, joint pain, fever |
0.97 |
| Hx gastro symp |
−0.50 |
Female, age 30+, cough |
−0.46 |
| Cough |
−0.04 |
Female, age 30+, runny nose |
−0.80 |
| Loss of appetite |
−0.07 |
Female, age 30+, headaches |
0.07 |
| Excess sweat |
−0.39 |
Female, age 30+, muscle aches |
−0.53 |
| Abdominal pain |
0.00 |
Female, Hx resp symp, wheezing |
2.00 |
| Fever |
−1.10 |
Female, cough, runny nose |
0.91 |
| White, female |
−0.86 |
Female, cough, difficulty breathing |
−1.14 |
| White, age 30+ |
0.98 |
Female, runny nose, headaches |
−0.14 |
| White, shortness of breath |
0.97 |
Female, runny nose, pinkeye |
1.16 |
| White, vomiting |
−1.45 |
Female, sore throat, muscle aches |
−0.70 |
| White, joint pain |
0.49 |
Female, sore throat, chills |
−3.41 |
| White, fever |
−0.32 |
Female, sore throat, fever |
2.16 |
| Female, Hx neuro symp |
−0.89 |
Female, headaches, abdominal pain |
−0.76 |
| Female, sore throat |
0.09 |
Female, headaches, pinkeye |
0.74 |
| Female, fatigue |
1.02 |
Female, fatigue, difficulty breathing |
1.34 |
| Female, shortness of breath |
−1.55 |
Female, chills, fever |
−0.21 |
| Female, pinkeye |
1.31 |
Age 30+, Hx resp symp, Hx inflam symp |
−0.19 |
| Female, numbness |
−1.17 |
Age 30+, Hx resp symp, sore throat |
−0.14 |
| Age 30+, Hx gastro symp |
−1.12 |
Age 30+, Hx resp symp, muscle aches |
−2.64 |
| Age 30+, fatigue |
−1.70 |
Age 30+, Hx resp symp, difficulty breathing |
−0.55 |
| Age 30+, shortness of breath |
−0.15 |
Age 30+, runny nose, headaches |
0.71 |
| Age 30+, fever |
−0.49 |
Age 30+, sore throat, joint pain |
0.07 |
| Hx resp symp, cough |
0.53 |
Age 30+, Headaches, loss of taste |
0.07 |
| Hx resp symp, chest pain |
−1.66 |
Age 30+, headaches, red rash |
−0.23 |
| Hx resp symp, joint pain |
0.03 |
Age 30+, Fatigue, muscle aches |
−0.10 |
| Hx neuro symp, diarrhea |
0.00 |
Age 30+, diarrhea, fever |
0.65 |
| Hx neuro symp, fever |
0.86 |
Age 30+, chest pain, fever |
0.49 |
| Cough, sore throat |
−0.68 |
Age 30+, vomiting, fever |
−0.14 |
| Cough, fatigue |
−0.46 |
Age 30+, pinkeye, red rash |
−0.79 |
| Cough, muscle aches |
1.52 |
Hx resp symp, cough, muscle aches |
−1.00 |
| Cough, loss of smell |
0.81 |
Hx resp symp, cough, fever |
0.25 |
| Runny nose, headaches |
−0.07 |
Hx neuro symp, diarrhea, difficulty breathing |
0.33 |
| Runny nose, fatigue |
−0.66 |
Hx neuro symp, diarrhea, fever |
0.00 |
| Runny nose, wheezing |
0.00 |
Cough, runny nose, loss of taste |
0.04 |
| Runny nose, fever |
−0.55 |
Cough, runny nose, loss of smell |
2.10 |
| Headaches, muscle aches |
0.54 |
Cough, runny nose, fever |
0.73 |
| Headaches, red rash |
−1.28 |
Cough, sore throat, headaches |
−0.64 |
| Headaches, fever |
0.40 |
Cough, sore throat, muscle aches |
−0.01 |
| Fatigue, diarrhea |
0.57 |
Cough, headaches, vomiting |
1.28 |
| Muscle aches, abdominal pain |
−0.96 |
Cough, chills, fever |
−0.15 |
| Muscle aches, wheezing |
−1.00 |
Cough, abdominal pain, loss of taste |
0.15 |
| Muscle aches, fever |
1.94 |
Cough, loss of taste, loss of smell |
0.33 |
| Chills, wheezing |
0.57 |
Cough, loss of taste, fever |
0.10 |
| Difficulty breathing, loss of appetite |
−0.39 |
Runny nose, sore throat, fever |
0.27 |
| White, female, Hx resp symp |
−0.19 |
Runny nose, headaches, muscle aches |
0.02 |
| White, female, headaches |
0.44 |
Runny nose, headaches, shortness of breath |
0.18 |
| White, female, fatigue |
−0.31 |
Runny nose, headaches, vomiting |
0.56 |
| White, female, diarrhea |
−0.85 |
Runny nose, muscle aches, diarrhea |
−2.04 |
| White, female, loss of smell |
−0.56 |
Runny nose, loss of appetite, wheezing |
−0.30 |
| White, female, fever |
0.00 |
Sore throat, headaches, fatigue |
0.42 |
| White, age 30+, cough |
−0.82 |
Sore throat, headaches, difficulty breathing |
2.93 |
| White, Hx resp symp, cough |
1.71 |
Sore Throat, muscle aches, fever |
−1.05 |
| White, Hx resp symp, loss of balance |
0.24 |
Sore throat, vomiting, fever |
−0.46 |
| White, Hx neuro symp, cough |
0.37 |
Headaches, fatigue, wheezing |
−1.02 |
| White, cough, headaches |
1.01 |
Headaches, muscle aches, vomiting |
0.33 |
| White, cough, loss of appetite |
0.77 |
Headaches, muscle aches, wheezing |
−0.68 |
| White, cough, fever |
0.02 |
Fatigue, muscle aches, abdominal pain |
−0.24 |
| White, sore throat, joint pain |
0.13 |
Fatigue, chills, wheezing |
0.90 |
| White, headaches, chills |
0.96 |
Fatigue, chest pain, wheezing |
1.45 |
| White, headaches, loss of taste |
0.77 |
Fatigue, vomiting, fever |
−0.07 |
| White, fatigue, chest pain |
1.54 |
Fatigue, abdominal pain, wheezing |
0.31 |
| White, fatigue, shortness of breath |
0.36 |
Muscle aches, loss of appetite, shortness of breath |
0.38 |
| White, fatigue, fever |
0.53 |
Muscle aches, chest pain, fever |
0.66 |
| White, muscle aches, chills |
0.91 |
Diarrhea, difficulty breathing, abdominal pain |
0.48 |
| White, chest pain, wheezing |
0.46 |
Diarrhea, abdominal pain, loss of taste |
0.03 |
|
|
Diarrhea, joint pain, fever |
−2.07 |
Abbreviations: gastro, gastrointestinal; Hx, history; inflam, inflammatory; neuro, neurological; resp, respiratory; RT-PCR, reverse transcription polymerase chain reaction; symp, symptom.
aTable 3 reports coefficients of ordinary regression of RT-PCR test results on variables, pair of variables, and triplet of independent variables that had nonzero coefficients in 95% of repeated LASSO regressions in phase 1 data (training set).
bCoef refers to regression coefficients.
Table 4 reports the percentage of variation in RT-PCR test results explained by the computerized symptom screening, at-home antigen tests, or the combination of 2, symptom screening and at-home testing. The first antigen at-home test explained 27.4% of the variation in RT-PCR test results. When the second antigen test was added in, the percentage of variation explained improved by 2.0%. In contrast, when the first antigen test was augmented with computerized symptom screening, the percentage of variation in RT-PCR test results explained improved by 6.9%, from 27.4% to 34.3%. When the first at-home test, second at-home test, computerized symptom survey, and vaccination status were used, the percentage of variation in RT-PCR test results explained increased to 47.6%, which is considered an excellent prediction of the variation in test results using McFadden statistic.16 Surprisingly, symptom screening and vaccination status (and their interactions) were statistically significant, suggesting that the first or second at-home tests were not informative.
Table 4. -
Value of At-Home Rapid Tests With and Without Symptom Screening
|
All Patients |
Vaccinated |
Not Vaccinated |
| Model |
Significant Variables |
Pseudo-R
2
|
Significant Variables |
Pseudo-R
2
|
Significant Variables |
Pseudo-R
2
|
| Y ∼ T1 |
T1 |
0.274 |
T1 |
0.278 |
T1 |
0.304 |
| Y ∼ T2 |
T2 |
0.14 |
T2 |
0.134 |
T2 |
0.163 |
| Y ∼ S |
S |
0.158 |
S |
0.112 |
S |
0.163 |
| Y ∼ (T1 + T2)2
|
T1, T2, T1, and T2 |
0.294 |
T1 |
0.280 |
... |
0.363 |
| Y ∼ (T1 + S)2
|
T1, S |
0.343 |
T1, S, T1, and S |
0.303 |
S |
0.567 |
| Y ∼ (T1 + T2 + S)2
|
T1, T2, S, T1 and T2 |
0.368 |
T1, S |
0.304 |
S |
0.712 |
| Y ∼ (T1 + T2 + S + V)2
|
S, V, S, and V |
0.476 |
|
|
|
|
Abbreviations: S, symptom screening; T1, first at-home test; T2, second at-home test; Y, RT-PCR test results.
An examination of the differences in predicting RT-PCR test results for vaccinated and unvaccinated individuals showed that all models were less accurate for the vaccinated study participants, presumably because vaccinated individuals have shown to report milder symptoms and less severe illness than the unvaccinated ones.17 For example, computerized symptom screening was 5.1% less accurate among vaccinated study participants. Similarly, a combination of at-home tests and computerized symptom screening was 26.4% less accurate among vaccinated study participants. The order of the accuracy of models across vaccinated and nonvaccinated patients did not change.
Table 5 reports coefficients from logistic regression of RT-PCR test results on individual symptoms, pairs of symptoms, and triplets of symptoms when combined with vaccination status and selective demographics, using phase 2 data set. Only statistically significant coefficients are reported here. The best predictors of positive RT-PCR test results among individual symptoms were loss of smell, vomiting, chills, cough, fatigue, and joint pain. Being 30 years or older was also a statistically significant predictor of a positive test result. Being vaccinated and reporting diarrhea were the least likely predictors of a positive RT-PCR test, with coefficients of −2.65 and −3.10, respectively.
Table 5. -
Predictors of
COVID-19 Test Results in Phase 2 Data
|
Coefficients of Nonzero Independent Predictors of RT-PCR Test Results |
|
Individual Symptoms |
Individual Symptoms, Pairs of Symptoms |
Individual, Pairs, and Triplets of Symptoms |
| Intercept |
−0.13 |
−0.12 |
−0.05 |
| Female |
−0.91 |
... |
... |
| Age 30+ |
1.79 |
0.43 |
0.63 |
| White race |
−0.41 |
... |
... |
| Abdominal pain |
−0.37 |
... |
... |
| Chills |
1.60 |
0.26 |
0.06 |
| Cough |
1.33 |
0.64 |
0.53 |
| Diarrhea |
−3.10 |
... |
... |
| Difficulty breathing |
1.14 |
... |
... |
| Excessive sweat |
0.01 |
... |
... |
| Fatigue |
1.19 |
0.41 |
0.40 |
| Joint pain |
1.30 |
... |
... |
| Loss of appetite |
0.33 |
... |
... |
| Loss of smell |
3.81 |
1.66 |
1.64 |
| Muscle aches |
−0.60 |
... |
... |
| Pinkeye |
−0.10 |
... |
... |
| Runny nose |
−0.97 |
... |
... |
| Vomiting |
2.23 |
0.43 |
0.71 |
| Wheezing |
0.91 |
... |
... |
| Vaccinated |
−2.65 |
−1.07 |
−0.93 |
| Female and fatigue |
... |
0.04 |
... |
| Female and vaccinated |
... |
−0.77 |
−0.91 |
| Age 30+ and chills |
... |
0.18 |
... |
| Age 30+ and fatigue |
... |
0.12 |
... |
| Age 30+ and joint pain |
... |
0.16 |
0.10 |
| White race and vaccinated |
... |
−0.07 |
−0.31 |
| Chills and cough |
... |
0.09 |
0.17 |
| Difficulty breathing and shortness of breath |
... |
0.21 |
0.01 |
| Female, White race and fatigue |
... |
|
0.23 |
| Chills, cough, and runny nose |
... |
|
0.18 |
Abbreviation: RT-PCR, reverse transcription polymerase chain reaction.
Table 6 reports performance metrics for the model, described in Table 5. We performed a 10-fold cross-validated analysis of phase 2 testing data (20%), which examined the accuracy of predication of COVID-19 diagnosis based on symptoms and vaccination status. Using individual symptoms, vaccination status and selective demographics (age and race) explained 42.8% (pseudo-R2 = 0.428) of variation in RT-PCR test results and reported area under the curve = 0.918.
Table 6. -
Accuracy of Predicting
COVID-19 RT-PCR Test Result 10-Fold Cross-Validated Performance Metrics
|
Individual Symptoms and Vaccination Status |
Individual, Pairs of Symptoms, and Vaccination Status |
Individual, Pairs, and Triplets of symptoms and vaccination status |
| Precision |
0.565 |
0.535 |
0.498 |
| Specificity |
0.667 |
0.667 |
0.667 |
| Sensitivity/recall |
0.760 |
0.800 |
0.663 |
| Area under receiver operating curve |
0.918 |
0.877 |
0.868 |
| McFadden pseudo-R
2
|
0.428 |
0.293 |
0.259 |
Abbreviation: RT-PCR, reverse transcription polymerase chain reaction.
Sensitivity of at-home testing
It is often difficult to interpret the McFadden percentage of variation explained. Policymakers and epidemiologists may be more familiar with sensitivity and specificity of diagnostic tests. Sensitivity and specificity are related concepts, and it is possible to increase sensitivity of a test by reducing its specificity. To make the measure of sensitivity more relevant, we examined it at fixed levels of specificity across various screening methods used in the study. Therefore, the increase in sensitivity is not due to loss of specificity but due to the increased accuracy of screening. To examine the sensitivity and specificity associated with computerized symptom screening, the phase 1 training data were used to identify an optimal cutoff point for symptom screening scores. Values above this cutoff point were considered to be positive for SARS-CoV-2 infection and below it, negative for SARS-CoV-2 infection.
Table 7 provides information on sensitivity and specificity of various screening methods, at nearly fixed levels of specificity. Of the 100 study participants who had tested positive for COVID-19, 58.7% were correctly identified by the first at-home test. The second at-home test did not increase the sensitivity of at-home testing. Using the 2 at-home tests and symptom screening did not result in higher sensitivity. In contrast, the addition of computerized symptom screening to the first at-home tests added an additional 11.1% to the sensitivity of predictions. This suggests that adding symptom screening to just one at-home test increases sensitivity of predictions with a negligible loss to specificity.
Table 7. -
Sensitivity of At-Home Tests and Symptom Screening at Fixed Specificity
|
RT-PCR Test |
|
|
|
|
| Model |
Neg |
Pos |
Sensitivity |
Specificity |
Precision |
Accuracy |
| Y ∼ T1 |
| Neg |
4277 |
279 |
0.587 |
0.989 |
0.892 |
0.915 |
| Pos |
48 |
396 |
|
|
|
|
| Y ∼ T2 |
| Neg |
4302 |
463 |
0.314 |
0.995 |
0.902 |
0.903 |
| Pos |
23 |
212 |
|
|
|
|
| Y ∼ S |
| Neg |
4113 |
600 |
0.111 |
0.95 |
0.261 |
0.838 |
| Pos |
212 |
75 |
|
|
|
|
| Y ∼ (T1 + T2)2
|
| Neg |
4277 |
279 |
0.587 |
0.989 |
0.892 |
0.935 |
| Pos |
48 |
396 |
|
|
|
|
| Y ∼ (T1 + S)2
|
| Neg |
4254 |
204 |
0.698 |
0.983 |
0.869 |
0.945 |
| Pos |
71 |
471 |
|
|
|
|
| Y ∼ (T1 + T2 + S)2
|
| Neg |
4254 |
204 |
0.698 |
0.983 |
0.869 |
0.945 |
| Pos |
71 |
471 |
|
|
|
|
Abbreviations: RT-PCR, reverse transcription polymerase chain reaction; Neg, negative, Pos, positive; S, symptom screening; T1, first at-home test; T2, second at-home test; Y, RT-PCR test results.
DISCUSSION
The second at-home antigen test has been proposed as a way of increasing the sensitivity of the first at-home test. Computerized symptom screening is also another way to improve the sensitivity of the first at-home test. Using 2 consecutive at-home antigen tests explained 29.4% of variation in RT-PCR test results. In contrast, using the first at-home test and the computerized symptom screening explained 34.3% of variation in the RT-PCR test. Therefore, adding computerized symptom screening is a better way to increase sensitivity of the first at-home test than adding another at-home test.
A major concern with at-home antigen test screening is that it could lead to many people mistakenly assume they do not have an active SARS-CoV-2 infection if tested negative, especially if asymptomatic. Those patients may not only receive poor treatment if developed symptoms and became ill but would also continue to spread the virus in the community. In our study, only 59% of the participants would have been correctly identified by the first at-home test, while 41% of tested individuals would have been missed. That is a large percentage of infected individuals going undetected. Combining the first at-home test with computerized symptom screening identified 70% of COVID-19-positive cases. This additional improvement in sensitivity occurred with a negligible loss of specificity (a drop of 0.006 points). These data suggest the importance of symptom screening in making at-home antigen tests more sensitive to detecting an active SARS-CoV-2 infection. One possible way forward is to rely on clinicians to interpret test results. Given the complexity of diagnosing COVID-19 from 50 different combinations of symptoms, it is unlikely that clinicians could provide an accurate diagnosis. This study shows that a computer aid can help clinicians improve the accuracy of the diagnoses when relying on patients' at-home test results.
When we combined vaccination status and the computerized symptom screening, the accuracy of predictions after the first at-home test increased significantly to yield a pseudo-R2 of 0.476, which is 20% higher than relying on the first at-home test alone. There is no technical reason why computerized symptom screening cannot also assess the vaccination status. One way to do so at home is through the phone. Alternatively, the symptom screening tool can be used by individuals without consulting their doctors if a website was made available to interact with the test takers of rapid antigen tests. Most people perform and interpret those tests at home, without consulting their doctor. So, it would likely be most feasible to allow end users to also consult a website to assess if their symptoms correspond to their test results. If the test results and symptom screening agree, the individual will have more confidence in the findings. They can then follow appropriate protocols, depending on whether the test result and the symptoms suggest a COVID-19 diagnosis. If the test results are not consistent with the symptoms, then the individual should visit a clinic, or get a confirmatory RT-PCR test.
The availability of smart phones, and related applications, suggests that symptoms, vaccination status, and rapid antigen test results can be entered into a phone and provide more accurate predictions. This study shows a way that rapid antigen test manufacturers can use to increase the accuracy of at-home tests. Although the manufacturers of rapid antigen tests can continue to claim emergency approval even after COVID-19 becomes endemic in the United States, they would need to present data supporting the accuracy of their tests to the FDA for a full nonemergency approval, at some point once the emergency status expires.
LIMITATIONS
This study had 3 main limitations. First, the data used to construct the model for predicting RT-PCR test results and for validating the accuracy of predictions were collected at 2 different periods during which (a) different variants of the novel coronavirus were predominant in the United States and (b) vaccination was not widely available during phase 1 data collection. Second, this study only examined one at-home antigen test, QuickVue by Quidel, which may have different sensitivity and specificity than some other at-home rapid antigen tests currently available. Third, the high attrition rate in phase 2 of the study puts into question the generalizability of our findings. Nonetheless, the findings of this study suggest that additional validation of at-home antigen tests combined with computer-aided symptom screening is warranted and could offer a game-changing solution to improve sensitivity of detecting infection.
REFERENCES
1. Schulte PA, Piacentino JD, Weissman DN, et al. Proposed framework for considering SARS-CoV-2 antigen testing of unexposed asymptomatic workers in selected workplaces. J Occup Environ Med. 2021;63(8):646–656. doi:10.1097/JOM.0000000000002269.
2. Dinnes J, Deeks JJ, Adriano A, et al.; Cochrane
COVID-19 Diagnostic Test Accuracy Group. Rapid, point-of-care antigen and molecular-based tests for diagnosis of
SARS-CoV-2 infection. Cochrane Database Syst Rev. 2020;8(8):CD013705. doi 10.1002/14651858.CD013705. Update in:
Cochrane Database Syst Rev. 2021;3:CD013705.
3. Deeks JJ, Dinnes J, Takwoingi Y, et al.; Cochrane
COVID-19 Diagnostic Test Accuracy Group. Antibody tests for identification of current and past infection with SARS-CoV-2. Cochrane Database Syst Rev. 2020;6(6):CD013652. doi: 10.1002/14651858.CD013652.
4. Kepczynski CM, Genigeski JA, Koski RR, Bernknopf AC, Konieczny AM, Klepser ME. A systematic review comparing at-home diagnostic tests for SARS-CoV-2: key points for pharmacy practice, including regulatory information. J Am Pharm Assoc (2003). 2021;61(6):666–677.e2. doi:10.1016/j.japh.2021.06.012.
5. BrĂ¼mmer LE, Katzenschlager S, Gaeddert M, et al. Accuracy of novel antigen rapid diagnostics for SARS-CoV-2: a living systematic review and meta-analysis. PLoS Med. 2021;18(8):e1003735. doi:10.1371/journal.pmed.1003735.
6. Prince-Guerra JL, Almendares O, Nolen LD, et al. Evaluation of Abbott BinaxNOW
rapid antigen test for
SARS-CoV-2 Infection at two community-based testing sites Pima County, Arizona, November 3-17, 2020. MMWR Morb Mortal Wkly Rep. 2021;70:100–105. doi:/10.15585/mmwr.mm7003e3external icon. Erratum:
MMWR Morb Mortal Wkly Rep. 2021;70:144. doi:10.15585/mmwr.mm7004a6.
7. Dinnes J, Deeks JJ, Berhane S, et al.; Cochrane
COVID-19 Diagnostic Test Accuracy Group. Rapid, point-of-care antigen and molecular-based tests for diagnosis of
SARS-CoV-2 infection. Cochrane Database Syst Rev. 2021;3(3):CD013705. doi:10.1002/14651858.CD013705.pub2.
8. Smith RL, Gibson LL, Martinez PP, et al. Longitudinal assessment of diagnostic test performance over the course of acute
SARS-CoV-2 infection. J Infect Dis. 2021;224(6):976–982. doi:10.1093/infdis/jiab337.
9. Alemi F, Vang J, Wojtusiak J, et al. Differential diagnosis of
COVID-19 and influenza. PLOS Global Public Health. 2022;2(7):e0000221.
10. Alemi F, Guralnik E, Vang J, et al. Guidelines for Triage of
COVID-19 Patients Presenting With Multisystemic Symptoms. Qual Manag Health Care. 2023;32(1 suppl):S3–S10. doi: 10.1097/QMH.0000000000000398.
11. Wojtusiak J, Bagais W, Vang J, Guralnik E, Roess A, Alemi F. The Role of Symptom Clusters in Triage of
COVID-19 patients. Special Issue: Diagnosis of
COVID-19 in the Community. Qual Manag Health Care. 2023;32(1 suppl):S21–S28. doi: 10.1097/QMH.0000000000000399.
12. Chen C-C, Lu S-C, Bai C-H, Wang P-Y, Lee K-Y, Wang Y-H. Diagnostic accuracy of SARS-CoV-2 antigen tests for community transmission screening: a systematic review and meta-analysis. Int J Environ Res Public Health. 2021;18:11451. doi:10.3390/ijerph182111451.
13. Wahid A, Khan DM, Hussain I. Robust adaptive Lasso method for parameter's estimation and variable selection in high-dimensional sparse models. PLoS One. 2017;12(8):e0183518. doi:10.1371/journal.pone.0183518.
14. Hughes G, Choudhury RA, McRoberts N. Summary measures of predictive power associated with logistic regression models of disease risk. Phytopathology. 2019;109(5):712–715. doi:10.1094/PHYTO-09-18-0356-LE.
15. Heinze G, Wallisch C, Dunkler D. Variable selection—a review and recommendations for the practicing statistician. Biom J. 2018;60(3):431–449. doi:10.1002/bimj.201700067.
16. McFadden D. Conditional logit analysis of qualitative choice behavior. In: Zarembka P, ed. Frontiers in Econometrics. New York, NY: Academic Press; 1974:105–142.
17. Tenforde MW, Self WH, Adams K, et al. Association between mRNA vaccination and
COVID-19 hospitalization and disease severity. JAMA. 2021;326(20):2043–2054. doi:10.1001/jama.2021.19499.
