Predictors and Impact of Self-Reported Suboptimal Effort on Estimates of Prevalence of HIV-Associated Neurocognitive Disorders

Abstract

Background: 

Prevalence estimates of HIV-associated neurocognitive disorders (HAND) may be inflated. Estimates are determined via cohort studies in which participants may apply suboptimal effort on neurocognitive testing, thereby inflating estimates. Additionally, fluctuating HAND severity over time may be related to inconsistent effort. To address these hypotheses, we characterized effort in the Multicenter AIDS Cohort Study.

Methods: 

After neurocognitive testing, 935 participants (525 HIV− and 410 HIV+) completed the visual analog effort scale (VAES), rating their effort from 0% to 100%. Those with <100% then indicated the reason(s) for suboptimal effort. K-means cluster analysis established 3 groups: high (mean = 97%), moderate (79%), and low effort (51%). Rates of HAND and other characteristics were compared between the groups. Linear regression examined the predictors of VAES score. Data from 57 participants who completed the VAES at 2 visits were analyzed to characterize the longitudinal relationship between effort and HAND severity.

Results: 

Fifty-two percent of participants reported suboptimal effort (<100%), with no difference between serostatus groups. Common reasons included “tired” (43%) and “distracted” (36%). The lowest effort group had greater asymptomatic neurocognitive impairment and minor neurocognitive disorder diagnosis (25% and 33%) as compared with the moderate (23% and 15%) and the high (12% and 9%) effort groups. Predictors of suboptimal effort were self-reported memory impairment, African American race, and cocaine use. Change in effort between baseline and follow-up correlated with change in HAND severity.

Conclusions: 

Suboptimal effort seems to inflate estimated HAND prevalence and explain fluctuation of severity over time. A simple modification of study protocols to optimize effort is indicated by the results.

INTRODUCTION

Estimates of the prevalence of HIV-associated neurocognitive disorders (HAND) are considerably varied. Between 22% and 84% of infected individuals meet criteria at any 1 time, with an average of about 50% based on larger consortium studies.^1–5 The majority of HAND diagnoses are mild, termed asymptomatic neurocognitive impairment (ANI) according to current research criteria.⁶ However, the inclusion of ANI in current diagnostic schema may have had the unintended consequence of high rates of false-positive diagnoses, thereby inflating HAND prevalence estimates.^7,8 This is primarily the result of the low threshold required to be considered neurocognitively impaired. Indeed, a significant percentage of healthy HIV-uninfected individuals with no known neurologic or psychiatric illness would meet criteria for ANI, save for the fact that they are not HIV infected.^8–12

Such findings do not invalidate ANI as a useful diagnosis. Indeed, a recent study found that individuals diagnosed with ANI at baseline progressed to symptomatic impairment faster than neurocognitively normal participants, as determined via self-report or performance-based measures.¹³ Although such findings underscore that ANI may be prodromal for more severe HAND, it does not quell the likelihood that many cases are misdiagnosed. Of note, that study did not report how many of the neurocognitively normal also progressed to ANI or how many of those with ANI recovered to a neurocognitvely normal status. This is important because changes in HAND status is common in both directions (recovery and worsening).^6,14,15 The explanation for this variability in HAND severity across visits remains incomplete.

One largely unexplored factor in fluctuating HAND severity and inflation of HAND prevalence is the influence of inadequate effort on psychometric testing outcomes. Such suboptimal effort, as defined here, occurs when research participants perform below their potential on neurocognitive tests. That is, because of 1 factor or a combination of several potential factors (eg, fatigue, apathy, distraction, boredom, intoxication), participants' scores do not reflect their true ability. This is different from feigned effort, in which performance is intentionally deficient with the goal of appearing cognitively impaired, although suboptimal effort has previously been used synonymously with feigned effort.^16–18 Feigned effort is a well-documented phenomenon in several patient populations, generally in the context of forensic or disability evaluations in which there is potential for secondary gain.^19–22 A variety of instruments, collectively termed performance validity tests (PVTs), have been developed for the purpose of identifying feigned effort. The sole effort study in the context of HAND used a PVT²³ among a research cohort. Not surprisingly, the participants almost all performed above the established cutoff for suspect effort. However, such tests are not valid for assessing suboptimal effort (as defined here) because their scoring criteria were developed for the detection of intentional response bias rather than insufficient implementation of cognitive abilities.

Anecdotal evidence suggests that a substantial portion of individuals enrolled in epidemiological studies may not put forth adequate and/or consistent effort on neurocognitive testing, an observation supported by empirical evidence in other research populations (eg, college students seeking course credit).^24–26 Such suboptimal effort may have substantial downstream effects on HAND prevalence estimates and our understanding of the variability of HAND severity over time. To investigate the role of suboptimal effort in HIV neurocognitive studies, we developed a novel visual analog measure that was integrated into the Multicenter AIDS Cohort Study (MACS) neurocognitive battery. Combined with the wide range of other information collected from MACS participants, this information was used to explore reason for, and predictors and consequences of, suboptimal effort. Our hypothesis are that suboptimal effort would contribute to overestimation of HAND rates and would help to explain variability in test performance over time.

METHODS

This study was conducted within the MACS, which has followed gay and bisexual men since the early 1980s. The visual analog effort scale (VAES) was developed by the authors specifically for investigating suboptimal effort. Visual analog scales are easy to administer and allow measurement of mental states in a continuous fashion. They demonstrate equivalent or better psychometric properties compared with ordinal scales, particularly with regards to assessing subjective states.^27,28 The VAES was administered to 935 participants at the conclusion of neurocognitive testing. Participants rated their effort on a line, with a range of 0%–100%. Those who reported <100% effort (ie, suboptimal effort) were directed to indicate the reason(s) for suboptimal effort. Fifty-seven participants completed the VAES at 2 consecutive visits, allowing for longitudinal analysis of effort and HAND severity in a subset of cases.

The following variables were included in our analysis of predictors and outcomes of suboptimal effort:

Neurocognitive Functioning

Participants complete a battery of neuropsychological tests as part of the standard study protocol.²⁹ This includes measures of working memory, learning, memory, executive functioning, motor functioning, and processing speed. T-scores were calculated using normative data with demographic corrections for age, education, and ethnicity. We examined both domain-specific T-scores and overall global neurocognitive functioning based on the average of the 6-domain T-scores.

HAND Severity

HAND status is determined for both HIV+ and HIV-uninfected participants via an algorithm developed by MACS investigators that adheres to the 2007 “Frascati” research criteria.^6,30 HAND status is based on neurocognitive test performance and self-reported deficits in activities of daily living.³¹ Participants are rated as neurocognitively normal, mildly impaired, moderately impaired, or severely impaired. The latter 3 correspond to ANI, minor neurocognitive disorder (MND), and HIV-associated dementia (HAD).

Substance Use

MACS participants complete a substance use questionnaire that assesses the frequency of use during the 6 months before the visit, including none, monthly, weekly, and daily. Participants report the frequency of use for alcohol, stimulants, marijuana/hashish, cocaine, opiates, or other recreational drugs (eg, 3,4-Methylenedioxymethamphetamine).

Depression

Depression severity is determined with the Center for Epidemiologic Studies Depression Scale (CES-D)³² as part of the standard MACS protocol. Scores on the CES-D are used as a continuous variable, with higher scores indicating greater degree of depression.

Memory Self-Rating

Participants are asked, “On a scale from 1 to 10, with 10 being normal for you, how would you rate your own memory ability now?”

Employment/Student Status

Participants indicate whether they are employed full time or part time, in school, or retired.

Statistical Analyses

Effort, as determined with the VAES, was used as a continuous variable. K-means cluster analysis categorized participants into 1 of 3 groups: high, moderate, or low effort, as described below. Differences in frequencies of reasons for suboptimal effort were compared between HIV status group and other categorical factors using χ² analysis, whereas continuous variables (eg, effort) were compared using analysis of variance. Linear regression used employed to determine predictors of suboptimal effort. Outliers (>3 SD above or below mean) with regards to neurocognitive domain scores were removed.

RESULTS

Suboptimal Effort Characteristics in MACS

The average VAES score was 91.4% (SD, 13), with a range of 20–100. There was no difference between HIV− (mean, 91.9; SD, 12.6) and HIV+ (mean, 90.9; SD, 13.6; F = 1.33; P = 0.249). Just more than half of the sample (51.7%) indicated suboptimal effort. Although there was not a statistically significant difference in effort across the 4 study sites (F = 1.69; P = 0.167), University of California, Los Angeles was observed to have the lowest effort (90.5%), whereas the site with the highest effort was Johns Hopkins (94.3%).

Table 1 displays reasons provided by the MACS participants who acknowledged suboptimal effort. For the 483 individuals who indicated suboptimal (<100%) effort, the most common reasons were “tired/fatigued” (43.4%) and “distracted/poor concentration” (36%). Less than 8% of those reporting suboptimal effort indicated “unmotivated” as a cause. This may seem contradictory, as effort and motivation appear synonymous. However, motivation is considered a different construct than effort.³³ Consider that one can feel unmotivated yet put forth full effort. As such, motivation is a feeling, whereas effort has the added dimension of action. More than 31% responded “other.” Common reasons provided among this subset were physical discomfort or illness (17%), memory difficulties (17%), boredom (15%), and anxiety (7%). Less common reasons (<5%) included hunger, learning disability, depression, testing environment, dislike for tests, and being in a hurry. There were no differences in reasons provided between serostatus groups.

TABLE 1.: Reasons for Suboptimal Effort

Effort Groups

K-means cluster analysis was used to create distinct effort groups. We settled on a 3-cluster solution, with groups defined as high effort (n = 707; mean effort = 97%; initial cluster center = 100%), moderate effort (n = 175; mean effort = 79%; initial cluster center = 60%), and poor effort (n = 53; mean effort = 51%; initial cluster center = 20%). We then examined how these effort groups differed. After correcting for multiple comparisons, groups differed in the cognitive domains of executive, speed, learning, and memory. Post hoc (Tukey B) analysis showed that all domains, the moderate and poor effort groups, differed from the high effort group but not each other. Groups also differed with regards to education, with poor effort group having significantly lower years of formal education compared with the moderate and high effort groups. Groups did not differ with regards to age or depression (CES-D). African Americans and Hispanics were overrepresented in the low and moderate effort groups as compared with white and “other” ethnicities (Table 2).

TABLE 2.: Demographic and Neurocognitive Comparisons Across Effort Groups

Effect of Suboptimal Effort on HAND Diagnoses

To begin to understand the effect of suboptimal effort on HAND prevalence, we examined the correlations between effort level and neurocognitive functioning. Pearson correlation analysis, after adjustment for multiple comparisons, revealed that effort was weakly correlated with all neurocognitive domain T-scores (R ≤ 0.23; P < 0.00001), with the exception of working memory and motor functioning (Table 3). Correlations between all neurocognitive domains were significant (P < 0.00001).

TABLE 3.: Correlations Between Effort and Neurocognitive Domain T-Scores

We then examined how the effort groups differed in frequencies of HAND diagnoses. Note that overall, 14.9% of the sample met criteria for ANI (15.3% of HIV+ and 14.5% HIV−), 11.2% for MND (13% of HIV+ and 8.9% of HIV−), and 2.7% for HAD (3.7% of HIV+ and 1.7% of HIV−). The poor effort group had greatest percent of ANI and MND diagnoses (25% and 33%, respectively) as compared with the high effort group (12% and 9%, respectively) and the moderate effort group (23% and 15%, respectively) (χ² = 58.12; P < 0.001). Unexpectedly, the poor effort group had no cases of HAD, whereas the moderate and high effort groups had 4.1% and 2.6% with HAD, respectively (Table 4).

TABLE 4.: HAND Ratings Across Effort Groups

Next, we looked at the relationship between change in effort and change in HAND severity between 2 study visits among 57 participants. This group had a mean age of 51 years (SD, 16), mean education of 14.8 years (SD, 2.7), and mean effort of 92.2% (SD, 10%). Two-thirds (66.7%) were HIV+, 58% reported suboptimal effort at baseline, and 65% were whites. The mean change in effort was +1.1% (SD, 7.7), with a range of between −20 and +20%. Twenty-two (39%) of participants indicated no change in effort, whereas 16 indicated a decrease and 19 indicated an increase. For HAND severity, 39 (68%) were stable between the 2 visits, 9 progressed to a more severe stage, and 7 improved to a less severe stage. The Spearman rank correlation between change in effort and HAND severity was significant (r = −335; P = 0.013). In a linear regression model, with change in HAND severity as the criterion variable and change in effort, race, CES-D (at second visit), cocaine use (at second visit), and self-reported memory ability (at second visit) as predictors, only change in effort remained in the final model (adjusted R² = 0.091; R² change = 0.110; F change = 5.579; significance of F change = 0.023). Note that none of the participants involved in the longitudinal analysis reported cocaine use during the previous 6 months.

Predictors of Effort Level

We then sought to determine the predictors of effort level based on the continuous score from the VAES. Based on initial correlations (not shown), the following predictor variables were included: age, education, employment status (employed or retired vs. unemployed and not retired), race (white vs. African American, regardless of Hispanic ethnicity), HIV status, depression (CES-D), alcohol use, cocaine use, and self-reported memory ability. In the final statistically significant model (adjusted R² = 0.079; R² change = 0.006; P = 0.028), decreasing effort was predicted by lower self-reported memory ability (β = 0.236; P < 0.001), African American race (β = −0.092; P = 0.012), and increasing frequency of cocaine use (β = −0.079; P = 0.028).

To further investigate the role of African American race and cocaine use in suboptimal effort, we examined frequencies of cocaine use between race categories and effort level between race categories (Table 5). Whites reported higher effort on the VAES (92.8%) as compared with African Americans (88.2%), regardless of Hispanic ethnicity (F = 21.39; P < 0.001). African Americans also reported more frequent cocaine use and were overrepresented in HAND diagnoses. To determine if the greater cocaine use among African Americans inflated this group's HAND prevalence, we removed all cocaine users and repeated the analyses. As shown in parentheses next to the aforementioned frequencies in Table 5, African Americans continued to be have greater rates of HAND, confirming that both cocaine use and ethnicity are independent predictors of effort. Finally, when only the high effort group was used in these analyses, the rate of HAND among African Americans decreased from 41% to 25%, whereas rates for whites decreased from 29% to 22%.

TABLE 5.: Racial Characterization in Regards to Cocaine Use and HAND Severity

DISCUSSION

Although ANI may be a useful diagnostic category that designates some individuals as being at the risk for functional decline,¹³ the low threshold required for diagnosis likely results in a large number of false-positive diagnostic errors.^7,8 Additionally, as investigated here, when prevalence rates are determined via research cohorts, there is threat of inflated estimates because of the suboptimal effort by study participants. Variable effort may also explain, in part, the apparent instability of HAND severity, such that a considerable proportion of individuals improve or decline across visits regardless of viral and immune factors.^6,30 In this study, we examined the phenomenon of suboptimal effort in one of the largest HIV study cohorts, the MACS. The data derived from our novel measure, the VAES, indicate that although more than 50% of participants reported suboptimal (<100%) effort, a much smaller number were considered to exert low (n = 53 or 6%) or moderate (n = 175 or 19%) effort. Still the effect of these cases on HAND prevalence estimates was remarkable, with 58% of the low effort group having mild-to-moderate HAND, compared with 38% of the moderate effort group and 21% of the high effort group. Importantly, although African Americans were disproportionally represented in both the low effort group and as having HAND (41% as compared with 25% of whites), when those who indicated low or moderate effort were removed, the rates of HAND among African Americans and whites nearly converged (25% and 22%, respectively). Finally, it is worth noting that the site with the best effort overall (Johns Hopkins) had the highest proportion of African Americans, and the site with the poorest effort overall (University of California, Los Angeles) had the lowest proportion. Johns Hopkin's protocol differs from that of the other 3 sites in that the neurocognitive testing is completed on a different day then the other study procedures. At the other study sites, the neurocognitive testing is generally the last of up to 3 hours of other procedures, including physical examination, blood draw, and filling out numerous questionnaires. These results indicate that suboptimal effort is a significant factor behind the disproportionate number of African Americans meeting criteria for HAND in the MACS and that a slight modification in study protocol can mitigate this.

When looking at effort overall, the strongest predictor was self-reported memory ability. That is, participants who rated their memory ability as poor also reported lower effort. This may reflect an attitude more than an accurate self-assessment, as the correlation between self-reported memory ability and effort was stronger than those with neurocognitive domain T-scores (results not shown). In this scenario, less effort is exerted because the individual does not believe that he or she will be able to perform well because of his or her memory deficit. Alternatively, this finding may reflect an actual neurocognitive impairment that is not adequately detected by the MACS neuropsychological test battery. Another predictor of suboptimal effort was education attainment. Note that neurocognitive test scores are standardized according to age, race, and education, so the findings here are not because of confounds inherent in the normative data. One interpretation is that education attainment is an indicator of overall attitude about testing or reflects the underlying motivation of participants to cognitive challenges.

The relationship between effort and HAND severity was significant. Furthermore, with regards to intraindividual variability in HAND severity over time, regression analysis revealed that it was only change in effort between baseline and follow-up visit that predicted change in HAND severity, depression, race, cocaine use, and self-reported memory ability (all significant predictors in cross-sectional analyses, save for depression) were not significant predictors. It is also notable that 33% were HIV uninfected, so viral and immune factors were unlikely to explain the change in HAND severity. Indeed, χ² analysis did not reveal significant differences in change in HAND severity or effort between HIV+ and HIV-uninfected participants (not shown). This finding underscores the significant contribution of effort to HAND variability over time. Furthermore, there has been recent interest in intraindividual neurocognitive variability in HIV as a behavioral marker of impending disability,³⁴ functional deficits,³⁵ and cognitive dyscontrol because of the combined effects of age and HIV in older (≥50 year old) adults.³⁶ Those studies did not consider effort, leaving this question to be addressed in the future. Indeed, studies in traumatic brain injury have indicated that intraindividual variability in neurocognitive impairment increases both as a function of neuropathology and of suboptimal effort.¹⁷

One might argue that effort is related to neurocognitive functioning, and as such, it is not surprising that those with low effort also have higher risk for HAND. However, the data do not support this. First, the correlations between VAES and neurocognitive domain scores, although statistically significant, were not strong and were similar across all effort groups. Second, if it were true, one would expect a relatively greater number of the low effort group to have severe HAND (ie, HAD). In fact, no one in the low effort group had HAD, whereas 4.1% of the moderate and 2.6% of the high effort group members had HAD. This unusual finding requires further exploration. One possibility is that individuals who are truly impaired (eg, with HAD) and aware of their deficits put forth better effort because they are more invested in learning about their true level of functioning, whereas individuals who do not perceive themselves to have cognitive deficits are less interested in what the testing may reveal.

Together with the predictors identified here, the reasons provided by suboptimal responders may enable modification of study protocols to ensure adequate effort. The most common reasons provided were tired/fatigued (43%) and poor concentration/distracted (36%). Considering the aforementioned difference in site protocols, it would be expected that these reasons would be less common at Johns Hopkins. Indeed, further examination of the data reveal that only 19% and 12% of respondents at Johns Hopkins indicated tired/fatigued and distracted/poor concentration as reasons for suboptimal effort, respectively. As such, modifying study protocols such that neurocognitive testing is completed when participants are fresh is strongly indicated by these results.

We acknowledge several limitations of this study. First, we did not use embedded PVTs in our primary analyses, despite the fact that several are included in the MACS neurocognitive battery. Embedded measures are portions of standard neurocognitive tests further developed for detecting feigned effort. These are essentially aspects of standard neurocognitive tests that are fairly easy to obtain perfect or near perfect scores on. Like standard PVTs, these embedded measures have been studied for their utility detecting feigned impairment among groups considered more likely to feign impairment (eg, Mild Traumatic Brain Injury patients in litigation) and among individuals instructed to complete the tests as if they were feigning impairment. Like other PVTs, these cutoffs would not be useful for detecting suboptimal effort because they were developed for identifying feigned vs. true effort.^37–40 It could be argued that using the scores in a continuous fashion, rather than cutoff scores, may be a better approach. However, our data (not shown) indicate that such scores correlate much more strongly with other neurocognitive tests score than effort as measured with the VAES. Therefore, neither explicit PVTs nor embedded measures are useful for assessing suboptimal effort in this context. Second, the VAES has not been tested for its validity or reliability. However, we point out that for adequate testing of this measure's psychometric qualities, a criterion measure of suboptimal effort is required. To our knowledge, no such measure exists. As such, this study has generated important validity and reliability data for the VAES, which we hope can be used for assessing suboptimal effort in other contexts. Third, the HAND classification system used by the MACS does not adequately assess for other causes of neurocognitive impairment. This is especially limiting considering that HAND is a diagnosis based on exclusion of other causes. However, like the MACS, other large cohort studies lack the resources to conduct the comprehensive diagnostic testing required to rule out other causes. Finally, there are other factors likely influencing effort or modifying neurocognitive performance as a function of effort. Examples include personality^41,42 and acculturation.^43,44 Thus, further investigation of factors that mitigate or augment effort in the context of research studies might consider factors not considered here.

One final point about inflated prevalence estimate: the finding reported here that almost as many HIV− and HIV+ MACS participants meet criteria for HAND underscores the poor specificity of current diagnostic criteria. Even when prevalence estimates are based only on those participants who reported full effort, 25.7% of HIV+ participants and 20.6% of the uninfected participants meet HAND criteria. This is especially troubling because prevalence estimates are generally cited from studies that use only HIV+ cases.^5,14,45,46 Although one more recent study included a HIV-uninfected comparison group,³ a large proportion of uninfected individuals still met criteria for HAND. Furthermore, the markedly higher rates of comorbidities (eg, substance use and depression) among the HIV+ sample complicate interpretability of that study. Considering that the MACS cohort possesses comparatively fewer comorbidities and that its HIV+ and HIV-uninfected participants are more similar with regards to comorbidities, the data presented here more accurately reflect the reality of the inadequacies of the current diagnostic schema. Fortunately, growing awareness of this has led some to develop alternative strategies for classifying HAND.⁴⁷

To summarize, suboptimal effort is a true phenomenon in HIV research studies that has subsequent influence on HAND prevalence estimates, public policy, and designation of resources. Suboptimal effort is also a factor behind variability of HAND severity over time. We have identified a very simple solution for improving effort. We strongly recommend that future determination of HAND prevalence rates and progression over time consider the phenomenon of suboptimal effort and employ alternative statistical methods.