Liu, Honghu PhD*; Miller, Loren G MD, MPH†; Hays, Ron D PhD*; Wagner, Glenn PhD‡; Golin, Carol E MD, MPH§; Hu, Wenhua MS‖; Kahn, Katherine MD*; Haubrich, Richard MD¶; Kaplan, Andrew H MD#; Wenger, Neil S MD, MPH*
Adherence to antiretroviral medication is critical to the effectiveness of HIV treatment,1-5 yet it is common for patients to receive irregular and incomplete drug dosing. Reasons for nonadherence are diverse and complex and include regimen complexity, substance abuse, inadequate belief in the efficacy of the medication, side effects, and the need for prolonged medication use.
It is essential for clinicians to identify nonadherent patients so they can intervene to augment adherence. Many adherence measurement tools have been developed, including self-report, pill count, an electronic monitoring system, pharmacy and administrative data, and serum drug concentrations.1,2,6 Each of these tools provides clinicians and investigators with challenges, however. For example, electronic pill caps yield an objective and relatively accurate adherence measurement, but these devices are expensive and labor-intensive, making them impractical for clinical settings. Pill counts, pharmacy records, and serum levels are methodologically sophisticated and expensive or time-consuming to analyze. Self-report measures are easy to administer, require no special equipment, and have been commonly used in clinical settings.7,8 Despite these advantages, self-reported adherence almost invariably overestimates a patient's true adherence and has low sensitivity for nonadherence.3,9,10 One study found the sensitivity of self-report for nonadherence, defined as taking <90% of prescribed medications, to be only 18%.11 Poor detection of nonadherence severely limits the utility of self-reported adherence measurement.
There are many reasons why self-reported adherence may be inflated.12,13 Patients may be embarrassed to reveal nonadherence to treatment. Admitting nonadherence may be perceived as a violation of the agreement with one's provider, with whom a patient wants to maintain trust. Furthermore, admitting nonadherence is inconsistent with the goal of the clinical encounter and may create feelings of guilt, leading to socially desirable unfounded claims of excellent adherence. Overestimation of adherence may be analogous to heavy alcohol users who underreport their alcohol consumption. In the latter case, indirect questions about alcohol use have been proven to be a valuable screening technique, and these are used clinically in the form of the Cut down, Annoyed, Guilty, Eye-opener (CAGE) questionnaire.14,15
A simple and accurate self-reported adherence measure that can be efficiently used in clinical settings would provide a major advance for clinicians and investigators. Such a tool would be inexpensive and practical. In addition, it could be used in resource-poor settings, where more complicated and expensive measurement tools, such as electronic devices, are not affordable.16-18
To explore the development of such measure, we investigated the discrepancy (differences) between patients' self-reported adherence and an objective adherence measure. We then evaluated the value of self-reported attitudinal measures to create calibration models that adjust patient self-reported adherence to represent the actual level of adherence to antiretroviral medications better. We used these models to identify a set of attitudinal items that may enhance the self-reported adherence and propose a new method to measure adherence using self-report in the clinical setting.
The patient sample was derived from 2 longitudinal HIV studies: the Adherence and Efficacy of Protease Inhibitor Therapy (ADEPT) and California Cooperative Treatment Group (CCTG) 578 investigations. The ADEPT investigation was a prospective, observational, single-center longitudinal study to measure patient adherence to antiretroviral medications objectively and to study the association between adherence and virologic outcomes. The ADEPT study collected detailed data on adherence and clinical and patient characteristics. It used multiple adherence measures, including an electronic measure (medication and event monitoring system [MEMS]; APREX Corporation, Freemont, CA),19 pill count, self-report, serum antiretroviral levels, and medication diaries. All subjects had started protease inhibitor (PI)-or nonnucleoside reverse transcriptase inhibitor (NNRTI)-containing combination therapy at baseline or within 3 months before study enrollment. Of the 144 subjects enrolled, 128 had analyzable adherence data. Patients were evaluated every 4 weeks for 48 weeks. Interviews at baseline and at weeks 8, 24, and 48 collected data on patient demographics, self-reported adherence, and clinical and other factors. A detailed description of the ADEPT study design can be found elsewhere.3,20,21
The CCTG 578 investigation was a prospective randomized trial of a cognitive-behavioral intervention for patients initiating or changing an antiretroviral regimen.22 A 3 × 2 factorial design was used, with the primary randomization assigning patients first to one of 2 cognitive-behavioral adherence interventions or usual care (1:1:1 randomization). A second randomization was to pharmacologic-assisted therapy with therapeutic drug monitoring or standard care on a 2:1 ratio within each of the 3 adherence intervention groups. Details of this study are published elsewhere.22 The CCTG 578 study enrolled 230 HIV-infected patients from 5 centers in California (one of which was also the site of the ADEPT investigation). Study subjects had similar demographic backgrounds as the ADEPT cohort. Survey and adherence data collected in the CCTG 578 study were similar to those collected in the ADEPT study, including longitudinal repeated-measures data on medication adherence and virologic outcomes. Entry criteria included more than 6 months of antiretroviral therapy, use of a PI-containing antiretroviral regimen, and HIV RNA level >400 copies/mL. Adherence to antiretroviral medications was measured by patient self-report, which asked about medication adherence over the 3 days before the survey, and by the MEMS. Self-report and MEMS adherence data were collected at weeks 4, 12, 24, and 48. Among the recruited 230 CCTG 578 study patients, 181 had analyzable adherence data.
Reconciliation of the ADEPT and CCTG 578 Data
Although the ADEPT and CCTG 578 investigations had similar study designs and data collection schemes, differences existed in data collection time points and the levels of categorizations of some survey items. To combine data from the 2 studies, we reconciled these differences. Specifically, the CCTG 578 investigation conducted adherence measures at weeks 4 and 12 (rather than at baseline and week 8 as in the ADEPT study). The week 4 CCTG 578 adherence measure was treated as a proxy for the baseline ADEPT survey, and the week 12 CCTG 578 adherence measure was treated as a proxy for the ADEPT survey at week 8. This enabled ADEPT and CCTG 578 subjects to have adherence measures at baseline and at weeks 8, 24, and 48. For discrepant response options to questions common to the ADEPT and CCTG 578 investigations, we recategorized categoric responses to have the same level of categorization between the 2 studies. For example, for the statement “If you don't take these medications exactly as instructed, the HIV in your body may become resistant,” the ADEPT survey had possible responses of “strongly disagree,” “disagree,” “uncertain,” “agree,” and “strongly agree” and the CCTG 578 survey had possible answers of “true,” “false,” and “don't know.” We regrouped “strongly agree” and “agree” as “true,” “disagree” and “strongly disagree” as “false,” and “uncertain” as “don't know.” Other items were merged similarly.
Self-reported adherence was collected by administering standardized medication adherence questionnaires asking the respondent about the past 3 days' (CCTG 578 study) or 7 days' (ADEPT study) adherence so as to reduce recall bias. After reconciling, variables for self-reported adherence were available at baseline and at weeks 8, 24, and 48 for each medication. We assume that past 3 days' adherence is a good approximation of past 7 days' adherence. To elicit adherence information, patients were asked the following question: “Many people don't take their medication perfectly all the time. Over the past 7 days, how many times did you miss a dose of [this medication]?” Using responses to this item, we calculated the mean adherence across the antiretroviral medications that a patient was taking. Adherence was considered to be a fraction of the doses of medications taken divided by the doses of medication prescribed. Adherence was expressed as a percentage and capped at 100%.
In addition to the adherence questionnaire, patients were asked questions that were directly or indirectly related to medication adherence. These questions are classified into 3 domains: (1) medication regimen descriptions (eg, “How many pills has your provider asked you to take each time?”), (2) medication regimen timing (eg, “How long have you been on antiretroviral medication?”), and (3) attitudinal factors (eg, “Taking HIV medication is too much trouble for what you get out of it.”).
Objective Adherence Measures
A “gold standard” adherence measure was computed for each patient. In the ADEPT study, detailed adherence information was measured with multiple tools, including self-reported adherence, the MEMS, and pill count. The MEMS is a pill bottle cap containing a microchip that records the date and time of each bottle opening.19 A MEMS cap was placed on the bottle containing each patient's newly started PI or NNRTI medication. We collected additional data such as use of pillbox and liquid medication to identify patients who were not using the MEMS, hence rendering the MEMS inaccurate. A pill count was calculated at each study visit, where the study nurse recorded the number of pills remaining in the patient's PI or NNRTI bottle(s). Adherence measures were summarized into 4-week periods. As previously reported,3 we created a composite adherence score (CAS) using a hierarchic algorithm, which combines adherence data from the MEMS and pill count. Because the MEMS is objective and most closely related to medication dosing, it was used as the backbone of the CAS. Before their use in computing the CAS, pill count adherence values were calibrated to the MEMS. Because the purpose of this investigation was to calibrate self-reported adherence, CAS values based on self-report adherence, included in former studies, were omitted. The algorithm to compute the CAS was published elsewhere.3,20,21
In the CCTG 578 study, MEMS adherence data were collected and supplemented with ancillary information from the patient survey, such as self-reports about the number of times that patients opened the bottle without removing a dose and number of times that patients removed more than 1 dose or took doses from another container. The adjusted MEMS adherence was then used as the gold standard for adherence measure for patients from the CCTG 578 investigation. Therefore, although the algorithms for objectively measured adherence in the CCTG and ADEPT investigations had differences, the MEMS was the backbone of the measure in both studies.
Face-to-face interviews were administered in the 2 studies at baseline and at weeks 4, 8, 12, 24, and 48 to collect information about patients' demographic and clinical characteristics and attitudinal and environmental responses. These measures included patient age, gender, education, income, race/ethnicity, antiretroviral history, injection drug use, children, trust in provider, CD4 cell count nadir, and peak viral load. Ten attitudinal measures focused on the following constructs (see Appendix for wording of items and response options):
* Self-efficacy for taking antiretroviral medications
* Support for taking antiretroviral medications
* Values of antiretroviral medications
* Relation of adherence and HIV resistance to antiretroviral medications
* Loss of control
Calibration Models and Cross-Validation Model Selection
Calibration models have been used in various areas to improve the accuracy and precision of measurement.23-26 Repeated measurements permit the development of multivariate calibration models. We used those models predicting objectively measured adherence to test whether multiple attitudinal measures corrected the bias from self-reported adherence. If Yi denotes the vector of objectively measured adherence for the i-th patient, we can write the repeated-measures model in a general form as follows:
Equation (Uncited)Image Tools
where n is the number of patients; Yi is an (ni × 1) vector of the objectively measured adherence for the i-th patient; ni is the number of measurements for the i-th patient over time, ranging from 1 to 3; Xi is (ni xp, with the first column being constant 1, the second column being time, the third column being self-reported adherence, and the remainder being patient characteristics and other measures) a vector of known predictors; α, the calibration coefficient, is a (p × 1) vector of fixed-effects parameters; and ϵi ∼ N(0, Σi).
In multivariate calibrations, calibration models aim to have minimal residual error so as to yield the best prediction for future subjects.27 To achieve this goal, cross-validation was used to search for optimal calibration models.28,29 This is done by splitting the given total sample size, N, according to some criterion, into 2 parts with sample size Nc for model construction and sample size Nv for validation with Nc + Nv = N. Because the available sample size from the ADEPT and CCTG 578 investigations is relatively limited, the most efficient cross-validation approach for the combined data is to combine estimation and validation into a single step, that is, to cross (ie, drop from parameter estimation) 1 subject at a time, to calculate the deleted residual, and then to repeat this process N times.30,31 The statistic that corresponds to the cross-validation approach with crossing 1 subject at a time is the predicted residual sum of squares (PRESS),32,33 which is defined as follows:
Equation (Uncited)Image Tools
Equation (Uncited)Image Tools
is the parameter when the i-th subject is deleted from the model fitting. Based on the PRESS statistics, different calibration models are ranked. The best calibration models are identified by smaller PRESS values.
The mean (SD) age of the patients was 38.5 (8.1) years, and 80% were male. Approximately half of the patients were Latino, 19% were African American, and 24% were white. At study entry, 39% were naive to antiretroviral medications. The median CD4 count was 94 cells/μL (range: 0-1573 cells/μL). Additional demographic and clinical characteristics are summarized in Table 1.
The objectively measured adherence of the cohort decreased significantly over time (P < 0.0001). Mean objectively measured adherence rates were 81%, 80%, and 74% at weeks 8, 24, and 48, respectively. Self-reported adherence remained nearly constant from week 8 to week 48, however, with mean values of 96%, 96%, and 95% for weeks 8, 24, and 48, respectively. Additionally, the variation for the objectively measured adherence was larger than that for self-reported adherence. The SD (coefficient of variation) ranged from 22.7 (0.28) to 24.9 (0.34) for objectively measured adherence and from 11.0 (0.12) to 15.8 (0.17) for self-reported adherence measures, respectively. Self-reported adherence was significantly higher than objectively measured adherence, with self-report adherence exceeding objectively measured adherence by 14.8% at week 8 to 17.4% at week 48 (Fig. 1).
There were significant variations in differences (δ) between objectively measured and self-reported adherence across the levels of response of attitudinal measures. For most of these attitudinal measures, the variations of the differences are complicated and do not show a simple and consistent trend across levels of response. Among the 10 attitudinal measures examined, 1 measure was significant (P < 0.05) at only 1 time point (“felt difficulties were piling up so high that you could not overcome them” at week 24) and 6 measures were significant at multiple time points or over time: “people who care about you encourage you to take your HIV medication” at weeks 8, 24, and 48 and over time; “HIV medication can help you to live longer” at week 8 and over time; “able to take medication the way your provider told you to” at weeks 24 and 48 and over time; “taking HIV medication is too much trouble for what you get out of it” at weeks 24 and 48 and over time; “felt confident about your ability to handle your personal problems” at weeks 24 and 48 and over time; and “felt that things were going the way you wanted” at weeks 24 and 48 and over time.
To construct calibration models, these 7 significant attitudinal measures, along with duration of treatment and self-reported adherence, were used to fit calibration models. With these 7 selected measures, a total of 27 = 128 possible calibration models were constructed (for the purpose of calibration, time on current antiretroviral medications and self-reported adherence were included in all models). The top 25 models had PRESS values ranging from 103,027 to 105,159 (Table 2).
According to PRESS results, the top 5 calibration models (having the smallest PRESS) included, in addition to self-reported adherence and regimen duration, the following attitudinal items: “able to take medication the way your provider told you to,” “HIV medication can help you to live longer,” “taking HIV medication is too much trouble for what you get out of it,” and “felt the things were going the way you wanted.” The top calibration model included 3 factors: time since start of current antiretroviral regimen, self-reported adherence, and “able to take medication the way your provider told you to.” Figure 2 shows the relation between self-reported adherence (x-axis) and the calibrated adherence (y-axis) using this calibration model with the minimal residual error. Although a few calibrated values are larger than the self-report values, most of the calibrated values are below the 45° line, indicating that there is generally shrinkage of self-reported adherence through the calibration model.
The sensitivities and specificities for predicting nonadherence of <90% or <95% using self-reported and calibrated adherence (using the top calibration model) are displayed in Table 3. This table shows that the sensitivities for detecting nonadherence using self-reported adherence were just 24% and 31% for detecting objectively measured adherence of <90% and <95%, respectively. In contrast, the sensitivities of calibrated adherence were much higher, ranging from 76% to 91% and from 74% to 89%, for the detecting <90% and <95% (objectively measured) adherence, respectively. The specificities were quite high for self-reported adherence and were low for calibrated adherence (see Table 3).
Using the top selected calibration model, Figure 3 shows a nomogram that demonstrates how the calibration system works. The x-axis is self-reported adherence, and the y-axis is calibrated adherence. The different lines correspond to 9 different scenarios about how long a patient has been taking antiretroviral medications (1, 6, or 12 months) and how well the patient can follow the way his or her doctor told him or her to do (none of the time, little/some of the time, or all the time). The top line, for example, is the calibration for patients who are taking antiretrovirals in the first month and can follow the way their doctors told them to do all the time. The full calibration model with the smallest PRESS is displayed in the footnote at the bottom of Figure 3. Based on the calibration coefficients, we can see that for each self-reported measure, approximately 50% of the self-reported reported value is kept in the calibrated adherence through the calibration model and that the remainder is adjusted by a constant of 21.77%, a penalty of 0.66% per extra month on taking antiretroviral medication (up to a total of 48 weeks after starting of a new regimen) and a 6.25% increase for each of the 2 levels higher in response to the question “able to take medication the way your provider told you to.” Based on the calibration coefficients, calibrated adherence values are higher as self-reported adherence increases, the patient's conviction that he or she is able to take medication as prescribed is stronger, and the time since regimen initiation is shorter. For example, for a patient with 85% self-reported adherence, in the sixth month for taking medication, and with the patient stating that he or she can take medication the way his or her doctor told him or her to all the time, the calibrated adherence value is 74.2% (the second line from the top). Based on the top calibration model, we also created a brief survey instrument for clinical use. This instrument can be used by the clinic front office or check-in personnel to ask these questions in the best calibration model (Fig. 4). The data elements from this form can be used with the nomogram to compute the calibrated adherence.
Inflated self-report adherence estimates hinder the ability of clinicians to identify patients' nonadherence to antiretroviral medication. This impairs clinicians' ability to act promptly to improve adherence. Our study demonstrates that survey items related attitudinally to adherence, in combination with self-report, can relatively simply produce an estimate of medication adherence that much more closely approximates objectively measured adherence. The efficiency of the survey approach makes it practical for use in busy clinics and, possibly, in resource-poor settings. The listing of antiretroviral medications also becomes part of the medical record and can identify patients who do not know their medications. This method fits in well with electronic records, which can compute calibrated adherence. Although these findings require testing and validating in other samples and settings, this tool may facilitate and improve HIV care.
The calibration models demonstrate that the accuracy of self-reported adherence is enhanced by survey items that tap into domains indirectly related to adherence. Among the predictors in the top 5 calibration models, the item related to time taking the current antiretroviral regimen reflects difficulties in sustaining adherence over time, an observation found in many cohorts.34,35 The attitudinal item asking about taking “medication the way your provider told you to” indirectly reflects adherence challenges that the patient could report without admitting nonadherence. The other attitudinal items reflect the patient's opinion about the effect of antiretroviral medications on prolonging life and the relative value of taking antiretroviral medications. Each of these attitudinal items modifies the meaning of the self-report adherence response. Although the proposed 3-item survey and others that could be formed from the calibration model presented are not a simple additive set of items, these questions can be easily completed during clinic intake with minimal disruption of clinical flow. The computation algorithm can be performed using the nomogram or could be programmed into a handheld electronic device. This screening tool has a high sensitivity for detecting nonadherence at critical levels (eg, 90% or 95%). Clinicians can then pursue a more detailed adherence assessment among positively screened patients.
There are limitations of the analyses and applications of our results. First, self-reported adherence was only measured intermittently and may not be able to capture the variation over time fully. Second, although the ADEPT and CCTG 578 studies had similar data collection designs, variation in data collection required reconciliation and some recategorization that might have affected our analyses. Third, subjects in the 2 cohorts, although derived from 5 different clinics, may not reflect patients in other settings and those who refuse to participate in clinical investigations. Finally, although we did not directly use the raw MEMS as the objective adherence measure, the MEMS was the backbone for calculation of the objective adherence measure for both studies. Because the MEMS tends to underestimate adherence, there could be some residual affect from the MEMS on the objective adherence measure.
In summary, we present a set of survey items and calibration models to screen for antiretroviral nonadherence using self-report of adherence and a few additional attitudinal items. The proposed items and calibration models merit formal evaluation in clinical settings.
The authors thank Victor Gonzales for technical assistance during the preparation of this manuscript.
1. Haubrich RH, Little SJ, Currier JS, et al. The value of patient-reported adherence to antiretroviral therapy in predicting virologic and immunologic response. California Collaborative Treatment Group. AIDS. 1999;13:1099-1107.
2. Paterson L, Swindells S, Mohr J, et al. Adherence to protease inhibitor therapy and outcomes in patients with HIV infection. Ann Intern Med. 2000;133:21-30.
3. Liu HH, Golin CG, Miller L, et al. A comparison study of multiple measures of adherence to HIV protease inhibitors. Ann Intern Med. 2001;134:968-977.
4. Knobel H, Carmona A, Grau S, et al. Adherence and effectiveness of highly active antiretroviral therapy. Arch Intern Med. 1998;158:1953.
5. Bangsberg DR, Hecht FM, Charlebois ED, et al. Adherence to protease inhibitors, HIV-1 viral load and development of drug resistance in an indigent population. AIDS. 2000;14:357-366.
6. Kalichman SC, Cain D, Cherry C, et al. Pillboxes and antiretroviral adherence: prevalence of use, perceived benefits, and implications for electronic medication monitoring devices. AIDS Patient Care STDS. 2005;19:833-839.
7. Waterhouse DM, Calzone KA, Mele C, et al. Adherence to oral tamoxifen: a comparison of patient self-report, pill counts and micro-electronic monitoring. J Clin Oncol. 1993;11:1189-1197.
8. Grymonpre RE, Didur CD, Montgomery PR, et al. Pill count, self-report and pharmacy claims data to measure medication adherence in the elderly. Ann Pharmacother. 1998;32:749-754.
9. Burney KD, Krishnan K, Ruffin MT, et al. Adherence to single daily dose of aspirin in a chemoprevention trial. An evaluation of self-report and microelectronic monitoring. Arch Fam Med. 1996;5:297-300.
10. Mason BJ, Matsuyama JR, Jue SG. Assessment of sulfonylurea adherence and metabolic control. Diabetes Educ. 1995;21:52-57.
11. Liu HH, Miller LG, Hays RD, et al. A comprehensive evaluation of survey questions for adherence to antiretroviral medications and exploratory analyses for identifying optimal sets of survey questions. AIDS Patient Care STDS. (In press).
12. Wagner J, Miller LG. Is the influence of social desirability on patients' self-reported adherence overrated? [letter]. J Acquir Immune Defic Syndr. 2004;35:203-204.
13. Nieuwkerk PT. Does electronic monitoring of adherence reduce social desirability on patients' self-reports of adherence? [letter]. J Acquir Immune Defic Syndr. 2004;36:880.
14. Beresford TP. Screening for alcohol abuse using the CAGE questionnaire. Am J Med. 1987;83:805.
15. Liskow B, Campbell J, Nickel EJ, et al. Validity of the CAGE questionnaire in screening for alcohol dependence in a walk-in (triage) clinic. J Stud Alcohol. 1995;56:277-281.
16. Oyugi JH, Byakika-Tusiime J, Charlebois ED, et al. Multiple validated measures of adherence indicate high levels of adherence to generic HIV antiretroviral therapy in a resource-limited setting. J Acquir Immune Defic Syndr. 2004;36:1100-1102.
17. Bangsberg DR, Ware N, Simoni J. Adherence without access to antiretroviral therapy in sub-Saharan Africa. AIDS. 2006;20:140-141.
18. Crane J, Kawuma A, Oyugi J, et al. The price of adherence: qualitative findings from patients purchasing fixed-dose combination generic HIV antiretroviral therapy in Kampala, Uganda. AIDS Behav. 2006;10:437-442.
19. View MEMS. User's Guide (Version 2.61). Union City, CA: APREX, Intelligent Technology for Medication Management; 1998.
20. Miller LG, Liu HH, Hays RD, et al. Knowledge of antiretroviral regimen and medication adherence. Clin Infect Dis. 2003;36:514-518.
21. Golin C, Liu HH, Hays R, et al. Attitude and demographic factors associated with antiretroviral adherence. J Gen Intern Med. 2002;17:1-10.
22. Wagner GJ, Kanouse DE, Golinelli D, et al. Cognitive-behavioral intervention to enhance adherence to ART: a randomized controlled trial (CCTG 578). AIDS. (In press).
23. Swierenga H, deWeijer AP, van Wijk RJ, et al. Strategy for constructing multivariate calibration models. Chemom Intell Lab Syst. 1999;49:1-17.
24. Borovnicar DJ, Stroud DB, Bines JE, et al. Comparison of total body chlorine, potassium, and water measurements in children with cystic fibrosis. Am J Clin Nutr. 2000;71:36-43.
25. Tosi S, Rossi M, Tamburini E, et al. Assessment of in-line near-infrared spectroscopy for continuous monitoring of fermentation processes. Biotechnol Prog. 2003;19:1816-1821.
26. Brien WF, Crawford L, Raby A, et al. In-house calibration of the international sensitivity index or calibration curve for determination of the international normalized ratio. Arch Pathol Lab Med. 2004;128:308-312.
27. Swierenga H, de Weijer AP, van Wijk RJ, et al. Strategy for constructing robust multivariate calibration models. Chemom Intell Lab Syst. 1999;49:1-17.
28. Stone M. Cross-validation choice and assessment of statistical predictions. Journal of the Royal Statistical Society, Series B. 1974;111-147.
29. Geisser S. The predictive sample reuse method with applications. Journal of the American Statistical Association. 1975;74:153-160.
30. Shao J. Linear model selection by cross-validation. Journal of the American Statistical Association. 1992;88:486-492.
31. Efron B. Estimating the error rate of a prediction rule: Improvement on cross-validation. Journal of the American Statistical Association. 1983;78:316-331.
32. Allen DM. Mean square error of prediction as a criterion for selecting variables. Technometrics. 1971;13:469-475.
33. Liu HH, Weiss RE, Jennrich IR, et al. PRESS model selection for repeated measures data. Journal of Computational Statistics and Data Analysis. 1999;30:169-184.
34. Rigsby MO, Rosen MI, Beuvais JE, et al. Cue-dose training with monetary reinforcement: pilot study of an antiretroviral adherence intervention. J Gen Intern Med. 2000;15:891-893.
35. Mannheimer S, Friedland G, Matts J, et al. The consistency of adherence to antiretroviral therapy predicts biologic outcomes for human immunodeficiency virus infected persons in clinical trials. Clin Infect Dis. 2002;34:1115-1121.
Table. APPENDIX Atti...Image Tools
© 2006 Lippincott Williams & Wilkins, Inc.