In 2013, close to 12 million people in the United States had abused an opioid such as a prescription pain reliever or heroin, 2.5 million were estimated to have opioid use disorder (OUD), and more than 1.4 million people had sought treatment for OUD (Substance Abuse and Mental Health Services Administration [SAMHSA, 2014b ]). Accidental poisonings are now the leading cause of accidental death (surpassing motor vehicle accidents) in adults aged 25 to 64 (Centers for Disease Control and Prevention [CDC, 2012a ]), and up to 61% of accidental poisonings are attributed to opioids (Rudd et al., 2016 ). Nonfatal opioid overdoses are believed to occur 3 to 7 times more frequently than fatal overdoses (Paulozzi et al., 2006; Coben et al., 2010; Dunn et al., 2010; Warner et al., 2011 ), and overdoses are estimated to account for more than 6000 emergency room visits per day (SAMHSA, 2013 ).
Unintentional opioid overdose has increased in several societal populations, including patients being treated for chronic pain, older patients, adolescents, and children (Cobaugh and Krenzelok, 2006; Paulozzi et al., 2006; Bailey et al., 2009; Coben et al., 2010; Dunn et al., 2010; Palmiere et al., 2010; Bohnert et al., 2011; Rosca et al., 2012; CDC, 2013 ). Overdoses result in lengthy hospitalizations for which costs are frequently absorbed by the public healthcare system, are more prevalent among Medicaid recipients, adversely affect low-income individuals (who experience a 2.1–5.7 greater risk of overdose relative to higher-income individuals), and are highest within rural settings, which already have limited financial resources (Hall et al., 2008; Paulozzi and Xi, 2008; Coolen et al., 2009 ). The CDC recently suggested that efforts towards preventing opioid overdose be intensified (Rudd et al., 2014 ).
There is a clear and urgent need to develop comprehensive efforts to address opioid overdose. One of the most highly publicized interventions is to distribute the opioid antagonist naloxone (Narcan), which can effectively reverse opioid overdose. There are substantial public health efforts to coordinate the distribution of naloxone to high-risk groups (CDC, 2012b; Walley et al., 2013; Clark et al., 2014 ); however, logistical barriers (such as increases in cost, laws preventing third-party medication administration by nonmedical personnel, need for in-person training sessions) may limit its widespread availability. Currently, the vast majority of opioid overdose prevention resources focus almost exclusively on naloxone administration (SAMHSA, 2014a ), which reduces the value of these resources in settings for which naloxone provision may not yet be available.
There is value in developing measures to assess knowledge of opioid overdose risk factors, which could be used to both complement or be used independent of naloxone-based interventions. There are 3 currently available opioid overdose knowledge assessments: the Opioid Overdose Knowledge (OOKS; Williams et al., 2013 ), which presents 36 true/false questions targeting factors, indicators, and behavioral responses to an overdose (including specific features of naloxone provision); the Opioid Overdose Attitudes (OOAS; Williams et al., 2013 ), which presents 28 Likert scale questions that focus on competency, concerns, and readiness to manage and/or intervene on an opioid overdose, and the Brief Overdose Recognition and Response Assessment (BORRA; Green et al., 2008 ), which presents 16 vignettes of possible overdose scenarios and is used to verify whether an individual can correctly determine whether naloxone should be administered. Though these scales are valuable for assessing learning after a naloxone administration intervention, they are not designed to facilitate brief educational interventions in the absence of naloxone. Further, each of these scales requires provider knowledge of opioid overdose response behaviors to interpret participant responses, and their lengths (28–36 questions, or the need to read and comprehend a vignette) may complicate their scoring and interpretation.
The current study describes efforts to develop a brief, knowledge-based measure that includes general information about opioids, overdose risk behaviors, and responses to an opioid overdose, to provide a standardized method to assess opioid overdose risk knowledge in settings for which naloxone dispensing and training is not yet available. To increase generalizability, this scale was developed in the context of both illicit and licit (patients prescribed opioids for chronic pain) opioid users. A brief knowledge-based measure would enable providers to quickly assess general knowledge gaps in their patients and to customize brief interventions for opioid overdose risk, similar to the Screening, Brief Intervention, and Referral to Treatment (SBIRT) interventions used for problematic drug and alcohol use. Though it is unlikely that a brief educational intervention will produce the high magnitude effect on overdose that is possible with provision of a pharmacotherapy such as naloxone, this will provide a brief and easy method to identify knowledge gaps and facilitate conversations between physicians, pain specialists, counselors, substance abuse treatment providers, and others with individuals who are using either illicit or licit opioids, and will serve as an additional resource to help combat the opioid overdose epidemic.
METHODS
General Study Design
This study was conducted in 2 phases among independent samples of illicit and licit opioid users. Phase 1 consisted of initial scale development and phase 2 consisted of scale confirmation. This study was approved by the Johns Hopkins University and University of Vermont Institutional Review Boards (IRBs), and waivers of informed consents were obtained for both sites. Participants consisted of 3 subgroups of opioid users (total N = 848; Table 1 ) and all data were collected between December 2013 and March 2015. All participants completed a brief demographic and opioid overdose questionnaire to characterize the study sample. The brief opioid overdose questionnaire asked whether the participant had ever overdosed, had ever witnessed an overdose, or had ever been trained to administer naloxone. To prevent biasing participant responses, overdose was not operationalized for participants. Demographic and drug use characteristics from the participants sampled in phases 1 and 2 are presented in Table 1 . The 3 samples differed in several important ways, including race, likelihood of being married and employed, and previous history of opioid overdose.
TABLE 1: Participant Characteristics
ANALYSES AND RESULTS
Phase 1: Initial Scale Development
Participants
Participants in sample 1 were illicit opioid users from Baltimore (N = 147). Participants were approached by a study staff member or responded to a flyer posted in their clinic or in the community to participate in the study. Participants who were confirmed to have OUD, were over 18, and were fluent in English were eligible for the study. The survey was provided as a self-report paper questionnaire, and study staff were available to assist participants with poor literacy. Participants were compensated up to $10 for survey completion.
Survey Questions
Participants completed an extensive 59-item self-report opioid overdose knowledge questionnaire that was derived from several peer-reviewed characterizations of opioid overdose knowledge among OUD patients (Dietze et al., 2006; Worthington et al., 2006; Baca and Grant, 2007; Sherman et al., 2008 ) (available in Supplemental Digital Content 1, https://links.lww.com/JAM/A43 ). Questions were rated on an ordinal scale and available responses were “True”, “False”, and “I Don’t Know” (to discourage random guessing and reduce the chance that participants may accidentally answer an item correctly) (Harris and Changas, 1994; Pennington et al., 2001; Herrmann et al., 2013 ).
Data Analysis
Responses to the overdose knowledge questions were dichotomized as correct or incorrect, and for analytic purposes items marked as “I Don’t Know” were categorized as incorrect. The initial set of 59 items generated 3 distinct factors (ie, general opioid knowledge, opioid overdose risk knowledge, and opioid overdose response knowledge), and responses were analyzed using item response theory. Two-parameter item response models were used to estimate a location term (β) and a discrimination term (α) for every response item, and a 2-parameter item response theory (2PL IRT) model with binary outcome variables was fit using confirmatory factor analysis (Brown, 2008 ). The latent response variable was scaled using marginal parameterization (Kamata and Bauer, 2008 ). Confirmatory factor analysis (CFA) was then conducted with robust weighted least squares estimation using the “lavaan” package (Rosseel, 2012 ) in the R statistical toolkit (R Development Core Team, 2011 ), and each of 3 proposed factors was fit separately using confirmatory factor analysis. The goal of this procedure was to identify items for which there was a range of item difficulties and room for improvement across a diverse sample of opioid users, to refine the item list to a shorter scale of well-functioning items, and to evaluate item performance across 2 unique opioid overdose risk groups (illicit and prescribed opioid users). Items with discrimination (α) less than 0.5 and with a location closer than 0.1 to both neighboring items were removed, refitting the model after any item was removed. Models for each factor were then combined to estimate a full, 3-factor confirmatory factor analysis with binary outcomes in sample 1. Each item was set to load only onto its intended factor, and IRT parameters for each item from this model were estimated. Finally, each factor was examined for question overlap, clinical utility, and generalizability, and items that were identified as having poor qualitative fit with other items within the factor were dropped.
Factor Construction
Analyses confirmed the 3 hypothesized, discrete factors within the item sets. Model fit was evaluated using the comparative fit index (CFI), the Tucker–Lewis index (TLI), and root mean square error of approximation (RMSEA). Consideration of a combination of model fit indices, with values of CFI and TLI ≥0.90 or higher indicating “acceptable” fit, and values of CFI and TLI ≥0.95 and RMSEA less than 0.06 indicating “good” fit, and minimizing both type I and type II error, even in models with small sample size (n ≤ 250; Hu and Bentler, 1999 ). The stand alone factor 1 (General Opioid Knowledge) included 9 items and had good model fit (CFI = 0.952, TLI = 0.936, RMSEA = 0.073 [95% confidence interval {CI} 0.033–0.109]) and 9 items were removed during factor construction and fitting. The stand alone factor 2 (Opioid Overdose Risk Knowledge) included 7 items and had excellent model fit (CFI = 0.980, TLI = 0.971, RMSEA = 0.047 [95% CI 0.000–0.103]) and 6 items were removed during factor construction and fitting. The stand alone factor 3 (Opioid Overdose Response Knowledge) included 10 items and had excellent model fit (CFI = 0.983, TLI = 0.978, RMSEA = 0.085 [95% CI 0.054–0.115]) and 7 items were removed during factor construction and fitting. Items in each factor were further assessed in terms of α and β, to identify the items in each scale with high α that best spanned the given range of β for that factor in the 26-item model. The final, combined, 3-factor model (Table 2 ) with 4 items per factor (12 items total) yielded good model fit (CFI = 0.964, TLI = 0.954, RMSEA = 0.052 [95% CI 0.000–0.083]). The resulting measure consisted of 4 items from each factor (12 items total) that demonstrated high factor loading in each sample, had high discrimination, covered a wide range of locations across the given construct, and were qualitatively sound in terms of importance in overdose knowledge, based on the existing literature.
TABLE 2: Percent of Subjects Endorsing
Phase 2: Scale Confirmation
Participants
The internal validity of this 12-item scale was then confirmed in samples 2 and 3. Sample 2 consisted of illicit opioid users from Baltimore, MD (n = 101) and Vermont (n = 98) (total N = 199). Sample 2 was identical in recruitment, eligibility criteria, and compensation to the illicit users in sample 1. Sample 3 (N = 502) consisted of participants who endorsed having chronic pain for ≥3 months and reported currently taking a prescribed opioid analgesic. Though some of these individuals also endorsed past 30-day drug use or misuse of their prescription (n = 111, 22.1% of chronic pain sample), the majority reported no recent drug use and using their prescription exclusively for pain management. These participants were recruited using online crowd-sourcing technology via Amazon Mechanical Turk (MTurk), an emerging form of participant recruitment that provides opportunities to sample populations who may not frequently attend a clinic (eg, chronic pain patients whose pain is well-controlled) and across large geographic regions, which reduces the potential for regional differences to differentially influence responding (Buhrmester et al., 2011 ). All participants responded to a request to complete a survey regarding “health behaviors,” and the survey was administratively restricted to individuals who resided within the United States and who had ≥80% approval rate from completion of previous MTurk tasks. Interested participants first completed a brief eligibility survey and the population being sampled was concealed to prevent participants from misrepresenting themselves to qualify for the primary survey. To be eligible, participants had to be over the age of 18, report having chronic pain for more than 3 months, endorse currently taking a prescribed opioid analgesic for pain, and be fluent in English. A total of 3157 individuals completed the eligibility survey and 502 met eligibility criteria and completed the study. Participants were compensated $3.00 via the MTurk Web site for their participation.
Survey Questions
To confirm the factor structure and internal validity of the measure, participants in samples 2 (illicit) and 3 (chronic pain) completed the 12 items that were identified in phase 1 of the study, which were treated as the final form of the measure and referred to as the Brief Opioid Overdose Knowledge (BOOK) questionnaire (Appendix). Overdose was defined at the beginning of the survey for chronic pain patients as “An overdose occurs when you take too high a dose of opioids, and it is not always fatal. Please answer these questions even if you are NOT SURE whether you ever overdosed on these medications, but know that you had a bad or scary experience from taking them.”
Data Analysis
Responses to the overdose knowledge questions were dichotomized as correct or incorrect, and items marked as “I Don’t Know” were categorized as incorrect. The questions administered during phase 1 were validated within the second opioid-using sample (sample 2) and the chronic pain sample (sample 3) using confirmatory factor analysis.
Factor Replication
Confirmatory factor analyses of the 12-item BOOK yielded good model fit in sample 2 (CFI = 0.957, TLI = 0.944, RMSEA = 0.041 [95% CI 0.000–0.065]) and sample 3 (CFI = .982, TLI = 0.977, RMSEA = 0.035 [95% CI 0.000–0.040]). Final item parameters are presented in Tables 2 and 3 . Table 2 presents the percent of participants in each sample that answered individual items correctly or as “I Don’t Know.” Table 3 presents the discrimination (α), location (β), threshold, and loading for each item.
TABLE 3: Factor Structure and Individual Items
BOOK Descriptive Analyses
Comparison of the 3 independent samples were conducted for descriptive purposes. For these analyses, the percent participants answering items correctly or endorsing “I Don’t Know” for each of the BOOK individual items were compared across the 3 independent samples using chi-square test for the individual items with z-scores for between-group comparisons. Comparisons of participant type (illicit drug users vs chronic pain patients) were then conducted to determine what characteristics and related correlates may underlie knowledge deficits. For these analyses, illicit drug users were collapsed across samples 1 and 2 and were compared with chronic pain patients (sample 3); to better differentiate these 2 groups, chronic pain participants were restricted to those individuals (n = 391) who did not report past 30-day drug use or misuse of their prescription. Associations between participant type and BOOK total and subscale scores were evaluated using independent-group t tests, and characteristics that may have contributed to differences in performance on the subscales (being an illicit opioid user, being over 30, being male, lifetime number of overdoses, history of witnessing an overdose, and being previously trained to deliver naloxone) were evaluated using linear regressions. Analyses were conducted using SPSS Version 21, and alpha was set at P < 0.05.
Results
Comparison of the 3 independent samples revealed considerable differences in performance across the individual items and overall scores (Table 2 ). Comparison of participant type (eg, illicit drug vs chronic pain groups) revealed significant differences in mean ratings for the total score (t [734] = 13.0, P < 0.001), with the illicit and chronic pain samples achieving mean (SD) 5.5 (3.1) and 3.0 (2.1) out of 12, respectively. This trend persisted with subscale 1, representing General Opioid Knowledge (t [734] = 15.6, P < 0.001), and subscale 3, representing Opioid Overdose Response Knowledge (t [734] = 14.8, P < 0.001); the groups did not differ significantly on subscale 2, representing Opioid Overdose Risk Knowledge (means presented in Fig. 1 ). Closer inspection of the relationship between scale scores and variables that may be associated with BOOK performance, assessed with linear regression, revealed that significantly better performance on the total score (R 2 = 0.21, F [6, 469] = 20.24, P < 0.001) was associated with being an illicit opioid user (b = 0.32, t [469] = 7.1, P < 0.001), being male (b = 0.10, t [469] = 2.26, P = 0.02), being over 30 years old (b = 0.17, t [469] = 4.2, P < 0.001), having experienced more lifetime overdoses (b = 0.12, t [469] = 2.76, P < 0.01), and having witnessed an overdose (b = 0.09, t [469] = 2.19, P = 0.03), whereas being trained to deliver naloxone did not contribute significantly to performance. A similar pattern was identified for the General Opioid Knowledge Subscale (R 2 = .26, F [6, 469] = 28.6, P < 0.001), where being an illicit opioid user (b = 1.23, t [469] = 9.73, P < 0.001), being over 30 (b = 0.14, t [469] = 3.47, P < 0.001), and having experienced more lifetime overdoses (b = 0.16, t [469] = 3.79, P < 0.001) were associated with significantly better performance, whereas witnessing an overdose, being trained to deliver naloxone, and being male were not associated with scores. Significantly better performance on the Opioid Overdose Knowledge Subscale (R 2 = 0.04, F [6, 469] = 3.5, P < 0.01) was associated with not being an illicit opioid user (b = 0.10, t [469] = 2.14, P = 0.03) and being younger than 30 (b = 0.13, t [469] = 2.82, P < 0.01); not being trained to deliver naloxone approached significance (b = 0.10, t[469] = 1.95, P = 0.051), and being male, having experienced more lifetime overdoses, and being trained to deliver naloxone had no significant contribution to performance. Finally significantly better performance on the Opioid Overdose Response Knowledge Subscale (R 2 = 0.22, F [6, 469] = 23.0, P < 0.001) was associated with being an illicit opioid user (b = 0.38, t[469] = 8.64, P < 0.001), being male (b = 0.10, t [469] = 2.40, P = 0.02), being over 30 (b = 0.12, t [469] = 3.01, P < 0.01), and having witnessed an overdose (b = 0.12, t [469] = 2.82, P < 0.01), whereas having experienced more lifetime overdoses and being trained to deliver naloxone had no significant association with performance.
FIGURE 1: BOOK Outcomes. Mean subscale response for illicit drug users vs. chronic pain patients. P values based upon results of independent groups t-tests, error bars represent SEM.
DISCUSSION
This study developed the BOOK questionnaire—a brief, internally valid measure to assess general gaps in knowledge in 3 areas that are hypothesized to contribute to opioid overdose (eg, opioid knowledge, opioid overdose knowledge, opioid overdose response knowledge). The final measure is a 3-factor, 12-item knowledge test that assesses items as “True,” “False,” and “I Don’t Know.” This BOOK questionnaire provides a brief and easy-to-administer method for quickly assessing patient knowledge of opioids, and can be used to complement existing naloxone interventions or in settings for which naloxone training may not yet be available. The BOOK questionnaire also adds to existing measures by focusing on general concepts, as opposed to previous measures that focused heavily on provision of naloxone for overdose reversal, and by verifying the responses in a large sample (N = 848) comprised of both illicit opioid users and patients prescribed licit opioids for chronic pain. Finally, the BOOK questionnaire is easy to score and is not expected to require training to administer and interpret results, which increases its potential for use in numerous settings and adds to the resources available to combat the opioid overdose epidemic.
From an initial 59 potential items derived from peer-reviewed characterizations of opioid overdose knowledge (Dietze et al., 2006; Worthington et al., 2006; Baca and Grant, 2007; Sherman et al., 2008 ) and evaluated using item response theory, analyses yielded a 12-item measure, with a 3-factor structure with strong model fit. The 3 factors represent 3 different dimensions of knowledge that may contribute to opioid overdose: General Opioid Knowledge, Opioid Overdose Risk Knowledge, and Opioid Overdose Response Knowledge. All 3 factors have strong internal validity, which supports their use either in combination with each other and with a total score, or independently to provide domain-specific assessments. Despite large differences in the demographic characteristics of these groups, the factor structure that was identified within the context of the first sample of illicit users was independently confirmed within the additional 2 samples of illicit and licit users, demonstrating the robustness of this measure.
This is the first opioid overdose risk measure to be developed within illicit and licit users. The items present general concepts designed to pertain to all opioid users, versus more nuanced concepts that would be relevant to only illicit or prescribed users (but not both). This increases the potential generality of the measure across settings (eg, primary care offices, schools, chronic pain providers, dentist offices, emergency rooms, jails/prisons, detoxification units, residential treatment programs) and patient populations. The goal of this measure is to provide a psychometrically sound method for assessing patient knowledge gaps to enable treatment providers to tailor a brief conversation with their patient in an effort to help reduce individual risk behaviors for experiencing an opioid-related overdose. Standardized knowledge measures can also facilitate development of educational interventions to reduce opioid overdose by providing a common metric across which clinical evaluations of prevention interventions can be compared and evaluated. Importantly, the BOOK questionnaire also provides opportunities to distinguish between answers that are incorrect and answers that are not known, which are qualitatively different domains of information. Responding to incorrect answers provides an opportunity to dispel myths or misinformation, whereas responding to answers marked as “I don’t know” provides an opportunity to educate patients about the topic. This feature may help to further enrich the provider–patient conversation.
Comparison of correct responses to the BOOK items across the illicit and chronic pain groups revealed significant differences on the total score, and also General Opioid Knowledge and Opioid Overdose Response Knowledge; no significant differences were observed regarding Opioid Overdose Risk Knowledge. Differences favored the illicit opioid group and suggested that being an illicit opioid user was associated with significantly greater performance on 3 of the 4 potential BOOK scales. These results may reflect the fact that most overdose reduction efforts are generally targeted towards illicit opioid users. The fact that chronic pain patients had comparatively lower levels of knowledge highlights the need for additional resources to be targeted towards this patient population. It should be noted, however, that the mean number of correct items was 50% or lower for each of the subscales, within both the illicit and chronic pain groups, and that performance was not significantly associated with having been trained to deliver naloxone for any subscale. These data support the development of non-naloxone-based educational curricula for persons who are exposed chronically to opioids more broadly.
This study has some notable limitations. First, the questions were derived through qualitative reports in the literature, so additional research is needed to verify the external validity of these questions. Second, the items are designed to cover general knowledge areas and therefore are not as specific as existing questionnaires regarding response to an opioid overdose. It is important to clarify that these items are not expected to replace existing measures or to be the most thorough assessment of content knowledge; rather, they have value for use as a quick method to gauge general patient knowledge, which could be used to facilitate conversations with patients, to help allocate overdose prevention resources, or to prioritize additional trainings. Further, because each subscale is internally valid, it is reasonable for providers to utilize the subscales of interest to them and to supplement the Opioid Overdose Response Scale with a more sensitive assessment of opioid overdose response behaviors if desired. Third, for brevity, limited demographic data were collected, which restricts analyses regarding how individual characteristics may impact outcomes. It also remains possible that patients may have experienced a nonfatal overdose they did not accurately identify as such, which could have impacted associations between overdose prevalence and BOOK outcomes. Finally, the chronic pain participants were recruited via crowd sourcing technology. This is an emerging form of participant recruitment that yields advantages over in-person forms of data collection because it can target populations who may not frequently attend the clinic (eg, patients whose chronic pain is well-controlled) and it enables sampling from large geographic areas, thereby reducing the opportunity for participant self-reports to be impacted by regional differences. Research has also validated the consistency between crowd-sourced responses and in-person clinic reports (Bartneck et al., 2015 ) and confirmed that data collected via crowd-sourcing conform to expected patterns (Boynton and Richman, 2014 ), further validating its use. Further, only 16% of the crowd-sourced participants who screened for this survey met eligibility criteria, which suggests the nature of the survey was successfully concealed to restrict false responding. Nevertheless, it remains possible that this population may differ from other chronic pain populations in ways we do not know.
CONCLUSIONS
In summary, opioid prescription rates continue to rise and there is a corresponding increase in the rate of opioid-related overdoses that is occurring across all segments of society. Currently, the only widespread treatment approach for opioid overdose prevention is the distribution of the opioid antagonist naloxone to high-risk populations; however, there are many other populations for whom the rate of opioid overdose is increasing, who are not yet receiving any systematic overdose prevention interventions. Existing measures focus on responses to overdose, and there are no standardized assessment measures that assess general gaps in knowledge for topics that could contribute to overdose risk. The BOOK questionnaire provides a brief and easy method to quickly assess knowledge gaps in 3 general content areas, which will enable providers to tailor informed discussions regarding opioid overdose risk with their patients, and will provide a metric from which different overdose intervention approaches may be compared and evaluated. Ultimately, this research provides an empirically-supported resource that can be used to help combat the opioid overdose epidemic and to advance the development of a more comprehensive approach for preventing opioid overdose.
APPENDIX SCORING INSTRUCTIONS FOR BRIEF OPIOID OVERDOSE KNOWLEDGE QUESTIONNAIRE
Opioid Knowledge Subscale:
Items 1, 2, 3, 4; sum number of TRUE items (range 0–4)
Opioid Overdose Knowledge Subscale
Items 5, 6, 7, 8; sum number of FALSE items (range 0–4)
Opioid Overdose Response Knowledge Subscale
Items 9, 10, 11, 12; sum number of TRUE items (range 0–4)
BOOK Total Score
Items 1, 2, 3, 4, 9, 10, 11, 12; sum number of TRUE items; items 5, 6, 7, 8 sum number of FALSE items (range 0–12)
ACKNOWLEDGMENTS
The authors would like to thank Eric Cunningham for his assistance with data entry and management.
REFERENCES
Baca CT, Grant KJ. What heroin users tell us about overdose.
J Addict Dis 2007; 26 4:63–68.
Bailey JE, Campagna E, Dart RC. RADARS System Poison Center Investigators. The underrecognized toll of prescription opioid abuse on young children.
Ann Emerg Med 2009; 53 4:419–424.
Bartneck C, Duenser A, Moltchanova E, et al. Comparing the similarity of responses received from studies in Amazon's Mechanical Turk to studies conducted online and with direct recruitment.
PloS One 2015; 10 4:e0121595.
Bohnert AS, Valenstein M, Bair MJ, et al. Association between opioid prescribing patterns and opioid overdose-related deaths.
JAMA 2011; 305 13:1315–1321.
Boynton MH, Richman LS. An online daily diary study of alcohol use using Amazon's Mechanical Turk.
Drug Alcohol Rev 2014; 33 4:456–461.
Brown TA. Confirmatory Factor Analysis for Applied Research. New York: Guilford Press; 2008.
Buhrmester M, Kwang T, Gosling SD. Amazon's Mechanical Turk: a new source of inexpensive, yet high-quality, data?
Perspect Psychol Sci 2011; 6 1:3–5.
National Vital Statistics System, National Center for Health Statistics, Centers for Disease Control and Prevention (CDC). 10 leading causes of death by age group, united states-2011. 2012a.
Centers for Disease Control and Prevention (CDC). Community-based opioid overdose prevention programs providing naloxone: United States, 2010.
MMWR Morb Mortal Wkly Rep 2012b; 61 6:101–105.
Centers for Disease Control and Prevention (CDC). Vital signs: Overdoses of prescription opioid pain relievers and other drugs among women: United States, 1999-2010.
MMWR Morb Mortal Wkly Rep 2013; 62 26:537–542.
Clark AK, Wilder CM, Winstanley EL. A systematic review of community opioid overdose prevention and naloxone distribution programs.
J Addict Med 2014; 8 3:153–163.
Cobaugh DJ, Krenzelok EP. Adverse drug reactions and therapeutic errors in older adults: a hazard factor analysis of poison center data.
Am J Health Syst Pharm 2006; 63 22:2228–2234.
Coben JH, Davis SM, Furbee PM, et al. Hospitalizations for poisoning by prescription opioids, sedatives, and tranquilizers.
Am J Prevent Med 2010; 38 5:517–524.
Coolen P, Best S, Lima A, et al. Overdose deaths involving prescription opioids among Medicaid enrolees: Washington 2004–2007.
Morb Mortal Wkly Rep 2009; 58 42:1171–1175.
Dietze P, Jolley D, Fry CL, et al. When is a little knowledge dangerous? Circumstances of recent heroin overdose and links to knowledge of overdose risk factors.
Drug Alcohol Depend 2006; 84 3:223–230.
Dunn KM, Saunders KW, Rutter CM, et al. Opioid prescriptions for chronic pain and overdose: a cohort study.
Ann Intern Med 2010; 152 2:85–92.
Green TC, Heimer R, Grau LE. Distinguishing signs of opioid overdose and indication for naloxone: an evaluation of six overdose training and naloxone distribution programs in the United States.
Addiction (Abingdon, England) 2008; 103 6:979–989.
Hall AJ, Logan JE, Toblin RL, et al. Patterns of abuse among unintentional pharmaceutical overdose fatalities.
JAMA 2008; 300 22:2613–2620.
Harris DK, Changas PS. Revision of Palmore's second facts on aging quiz from a true: false to a multiple-choice format.
Educ Gerontol 1994; 20:741–754.
Herrmann ES, Heil SH, Sigmon SC, et al. Characterizing and improving HIV/AIDS knowledge among cocaine-dependent outpatients using modified materials.
Drug Alcohol Depend 2013; 127 (1–3):220–225.
Hu L, Bentler P. Cut-off criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives.
Struct Equation Model 1999; 6 1:1–55.
Kamata A, Bauer DJ. Note on the relation between factor analysis and item response theory models.
Struct Equation Model 2008; 15:136–153.
Palmiere C, Staub C, La Harpe R, et al. Parental substance abuse and accidental death in children.
J Forens Sci 2010; 55 3:819–821.
Paulozzi LJ, Xi Y. Recent changes in drug poisoning mortality in the United States by urban-rural status and by drug type.
Pharmacoepidemiol Drug Safety 2008; 17 10:997–1005.
Paulozzi LJ, Budnitz DS, Xi Y. Increasing deaths from opioid analgesics in the United States.
Pharmacoepidemiol Drug Safety 2006; 15 9:618–627.
Pennington HR, Pachana NA, Coyle SL. Use of the facts on aging quiz in New Zealand: validation of questions, performance of a student sample, and effects of a don’t know option.
Educ Gerontol 2001; 27:409–416.
R Development Core Team. R: A language and environment for statistical computing. Vienna, Austria: The R Foundation for Statistical Computing; 2011. Available online at
http://www.R-project.org . Accessed July 11, 2016.
Rosca P, Haklai Z, Goldberger N, et al. Mortality and causes of death among users of methadone maintenance treatment in Israel, 1999–2008.
Drug Alcohol Depend 2012; 125:160–163.
Rosseel Y. Iavaan: an R package for structural equation modeling.
J Stat Softw 2012; 48 2:1–36.
Rudd RA, Paulozzi LJ, Bauer MJ, et al. Increases in heroin overdose deaths: 28 states, 2010 to 2012.
MMWR Morb Mortal Wkly Rep 2014; 63 39:849–854.
Rudd RA, Aleshire N, Zibbell JE, et al. Increases in drug and opioid overdose deaths: United States, 2000–2014.
MMWR Morb Mortal Wkly Rep 2016; 64 (50–51):1378–1382.
Sherman SG, Gann DS, Scott G, et al. A qualitative study of overdose responses among Chicago IDUs.
Harm Reduct J 2008; 5:1–5.
Substance Abuse and Mental Health Association. Highlights of the 2011 drug abuse warning network (DAWN) findings on drug-related emergency department visits. 2013. Available at:
http://www.samhsa.gov/data/2k13/DAWN127/sr127-DAWN-highlights.htm . Accessed July 11, 2016.
SAMHSA Substance Abuse and Mental Health Services Administration (SAMHSA). Opioid overdose prevention toolkit: updated. 2014a. Available at:
http://store.samhsa.gov/product/Opioid-Overdose-Prevention-Toolkit-Updated-2014/SMA14-4742 . Accessed July 11, 2016.
SAMHSA Substance Abuse and Mental Health Services Administration (SAMHSA) Results from the 2013 National Survey on Drug Use and Health: Summary of national findings No. NSDUH Series H-48, HHS publication No. (SMA) 14–4863). Rockville, MD: HHS; 2014b.
Walley AY, Xuan Z, Hackman HH, et al. Opioid overdose rates and implementation of overdose education and nasal naloxone distribution in Massachusetts: Interrupted time series analysis.
BMJ (Clinical Research Ed) 2013; 346:f174.
Warner M, Chen LH, Makuc DM, Anderson RN, Miniño AM. Drug poisoning deaths in the United States, 1980–2008. NCHS data brief, no 81. Hyattsville, MD: National Center for Health Statistics; 2011. Available at:
http://www.cdc.gov/nchs/products/databriefs/db81.htm . Accessed July 11, 2016.
Williams AV, Strang J, Marsden J. Development of opioid overdose knowledge (OOKS) and attitudes (OOAS) scales for take-home naloxone training evaluation.
Drug Alcohol Depend 2013; 132 (1–2):383–386.
Worthington N, Markham Piper T, Galea S, et al. Opiate users’ knowledge about overdose prevention and naloxone in New York City: a focus group study.
Harm Reduct J 2006; 3:19.