Second Round of Testing and Revisions
The second round of cognitive interviewing focused on expanding participant diversity and testing both English (n = 48) and Spanish (n = 47) versions (total n = 95). Online testing in English was completed with 236 participants (qualitative testing n = 159, quantitative testing n = 77) (Table 3). There were no new substantial changes in items that emerged from online qualitative testing in the second round. See Table 4 for estimated completion times. Review of the data led to very minor recommendations involving changing wording and adding clarifying language to some of the questions.
The Yield of this Development Process: Final Revisions to Survey Content
A summary of the over 40 recommended changes based on English language testing is included in Table 5. Minor changes consisted of small edits in phrasing of a question or response options (e.g., converting questions that were originally administered by interviewers in the parent survey to a format appropriate for self-administration). Major changes included more substantive modifications to a question or response options when noteworthy concerns were expressed by participants during qualitative testing. For example, the questions about race, ethnicity, and gender that were included in the first round of qualitative testing were modified and retested in the second round of testing based on input from participants and key Program stakeholders. These modifications led to refinement of our approach to race and ethnicity. We ultimately leveraged extensive testing completed by the US Census Bureau, which found that a combined question for capturing participant-reported race and ethnicity was the strongest approach for gathering complete and accurate data.37 The recommendations for changes in the Spanish language version included minor changes that implemented more simplified and common terminology. In addition, testing Spanish-language materials resulted in feedback and recommendations that went beyond translation, and included conceptual changes to questions to make them more understandable and accessible to their specific communities.
We explored concerns from members of the Program, such as about asking potentially sensitive questions regarding topics like gender identity and sexual orientation. Our testing did not confirm anticipated concerns but instead found an appreciation of being asked these questions among participants, including those self-identifying as sexual and gender minorities.
All questions were finalized by the committees and are summarized with their original instruments in eDocuments 1–6; http://links.lww.com/EDE/B518. The summary of iterative changes for both English and Spanish are summarized in eTable 4; http://links.lww.com/EDE/B518.
Exploratory Factor Analysis
Although much of the data in the three surveys were not amenable to further statistical analysis, an exploratory factor analysis was appropriate for two subcomponents (i.e., PROMIS, Brief Health Literacy Screen) of the English version of Overall Health. This factor analysis showed three distinct factors with coefficients alpha being 0.92 for General Physical Health, 0.81 for Emotional Health, and 0.55 for Health Literacy (eTable 5; http://links.lww.com/EDE/B518).
We created an iterative process leveraging diverse experts to develop and refine materials for collection of participant-provided information for All of Us that is applicable to a diverse audience, leverages existing validated surveys, and supports English and Spanish. We initially launched three surveys and are following this model for ongoing development of future surveys in All of Us.
Other large consortia such as the Million Veterans Program38 and the UK Biobank9,10 have included survey materials as a core data component; however, approaches for combining survey items from multiple sources in the context of a large research program are sparse. We learned several lessons useful for future Program development, as well as others undertaking similar work. First, we discovered generally minor issues with clarity and sensitivity for some module questions drawn from previously validated survey instruments. Although questions from certain validated instruments, such as the PROMIS Global Health Scale39 and the Brief Health Literacy Screen,40 performed well in our testing (eTable 3; http://links.lww.com/EDE/B518), we implemented minor modifications for other items (eTable 4; http://links.lww.com/EDE/B518). Modifications included adding examples to clarify a question (e.g., providing the number of cigarettes in a pack); new response options (e.g., e-cigarettes); and explanatory text before some items. As many of the module questions are derived from existing national surveys, we wanted to avoid substantial wording revisions, as this would compromise established scientific validity. Instead, we targeted all newly created explanatory text to the fifth-grade level and focused on identifying areas within the question text where minor revisions could lead to major improvement in readability. Second, we explored potential issues of concern to All of Us. In fact, similar to other findings,41 participants expressed appreciation for the Program’s recognition of the importance of asking about gender identity and sexual orientation, which initially concerned members of the Program. Third, we successfully leveraged prior extensive participant testing within national programs, such as the US Census,37 as our findings echoed the Census Bureau’s observations that a combined approach for querying race and ethnicity is more aligned with the way participants identify themselves and, thus, allows gathering more granular data. Fourth, we found that many validated questions from studies that may have been developed for a specific population, such as the California Teachers Study,42 only required minor wording changes, but otherwise performed well in a more diverse population. Finally, collaborating with consortium experts led to substantial improvements in integrating survey materials into the larger context of the Program. This collaboration led to a volume of survey items reasonable to ask participants to complete at enrollment, within the larger scope of enrollment activities such as consent and physical exam.
Our experience emphasizes the value of systematically vetting multiple languages to ensure optimum survey deployment, as well as the importance of testing with various dialects of a specific language. Creating and assessing surveys in both English and Spanish led to improved clarity, although maintaining concordance between translations. Testing in both languages ensured alignment in understandability and accessibility for the English and Spanish versions. This experience continues to inform the consortium’s work regarding the complexities of testing surveys in different languages. The translation process included review by members from multiple Spanish speaking regions to develop surveys understood by Spanish speakers from different regions. Instituting multiple versions of Spanish surveys is worth consideration; however, we did not find evidence within our interviews with participants that indicated this need.
We learned multiple lessons about cognitive interviewing strategies in this population. First, this testing process yielded valuable lessons learned regarding strategies for overcoming challenges in recruiting underrepresented or hard to reach populations. Over time, the Pilot team expanded recruitment to include methods such as in-person recruitment within the community, facilitating increased enrollment of harder-to-reach populations such as those without internet access, those with lower educational attainment, racial/ethnic minority populations, and Spanish-speaking participants. This face-to-face community interaction was an effective method to recruit certain populations that were more likely to engage in research opportunities through a trusted and familiar entity such as an established community organization. As echoed by others these methods require more time, but are critical to ensure inclusion of diverse populations.43–45 Second, cognitive interview probes authored by content experts allowed a deeper understanding of what needed to be explored to ensure accurate answers. Third, using a web application to perform interviews over the internet provided a cost-effective way to interview people who were not located locally. Fourth, our sample size was large, and we achieved saturation before reaching our full sample size. Because enrollment of underrepresented populations is a major aim of the All of Us Research Program, we wanted to be thorough in our attempts to include key populations in this component of the program planning. Future efforts of this type will likely include smaller enrollment targets. However, based on our experiences, we feel that the size and demographic characteristics of the sample are important and should be tailored to best fit the scope and goals of the project. For example, researchers interested in exploring readability and comprehension alone may reach saturation with a small sample. In instances where sensitivity and individual perspectives are also being sought, researchers may want to consider a larger and more diverse sample size.
We acknowledge several limitations: (1) Our testing did not include all populations underrepresented in biomedical research. Although we included very important populations, including sexual and gender minorities, those of low socioeconomic status, and Spanish-only speakers, gaps may require additional testing with other populations. (2) The speed at which initial survey development work proceeded limited our ability initially to include individuals from certain key groups, such as lower educational attainment. However, this was addressed during online testing and out second round of cognitive interviews. (3) The validity of combining questions from existing instruments was not thoroughly tested. To mitigate this, we intentionally minimized changes from existing instruments and executed entire scales or sets of questions about a topic area from a single instrument. (4) E-mail communication was the main method of contact for study recruitment, potentially limiting our reach of those less comfortable with technology. (5) Small sample size and potential lack of representativeness limits generalization of our exploratory factor analysis. (6) Online surveying poses an inherent risk, however small, of receiving a response from someone other than the intended participant. Finally, (7) we tested primarily on a computer platform and not on different technology platforms such as smartphones. Our plan will be to test future surveys on different platforms.
Deployment and Future Directions
To achieve the ambitious scope and scale of All of Us, surveys will need to be modular to not overwhelm participants, accessible on a digital platform so surveys can be completed in a variety of settings, available to be completed at a participant’s own pace, and engaging, so participants continue to contribute after initial enrollment in the Program.
Building upon this successful process for development of these initial materials, All of Us is developing other surveys (Table 2). Future areas of interest for the program will include rollout of these surveys, engaging participants to complete these additional surveys, and repeat administration of some surveys to reflect participant changes over time. As the Program evolves, survey development will continue to be a core activity. As described above, the marriage of scientific value, engagement, and participant experience will remain an important consideration for future work in this area. Future testing and integration with other sources of information, such as EHR data, mobile sensors, and a range of technology platforms, genetics, and physical measurements, will help enhance the value and completeness of All of Us data for future hypothesis exploration. Further validation of our findings related to the surveys in the launch of the Program will be needed to ensure what we found in our initial testing holds true for the larger cohort. Translation into other languages and the testing of these translations will also be an important area of activity as the Program grows.
Participant-provided information is a critically important part of the data that will power All of Us. The survey questions and response options must be carefully documented and communicated to researchers in a way that makes these data accessible and easy to integrate into external aspects of clinical research. An All of Us survey codebook is under development and is being designed to map All of Us survey items to standard vocabularies that can help align these data with EHR data where possible (e.g., for diagnoses). Finally, All of Us surveys reflect a mixture of questions from validated instruments in the public domain, accompanied by supplementary; questions that are not currently publicly available. All of Us will publish these surveys and their metadata, as they become available, at http://researchallofus.org.
This flexible process combined multidisciplinary expertise with Program leadership input and proven methods to create and refine surveys that are appropriate for use in the diverse participant population of All of Us. The process, which was generalizable across multiple survey domains, formed a firm roadmap for the development and testing of future materials. Other large consortia that target a diverse population in multiple languages could employ this process to create surveys that supplement other data sources, such as genetics and EHR data.
Prioritization and generation of All of Us surveys will continue in close collaboration with representatives across the NIH Institutes and Centers to ensure item integrity and scientific validity and help ensure that questions reflect the mission of the NIH to improve health outcomes broadly. Other surveys may be driven by participant interest. Participant engagement and experience will also be important components of future development. Gathering and incorporating participant-provided information in a systematic way will enhance the scientific validity and breadth of information obtained from All of Us, leading to exciting new advances in the era of precision medicine.
RESEARCH ETHICS AND INFORMED CONSENT
The research assessments described in this manuscript were approved by the Institutional Review Board of the All of Us Research Program. Those participants that completed cognitive interview gave oral consent to do so.
We wish to thank our participants who have joined All of Us and contributed to participant-provided information; helped refine early materials; engaged in the development and evaluation of the surveys, and provided other ongoing feedback. We thank the countless other coinvestigators and staff across all awardees and partners without which All of Us would not have achieved our current goals. All of Us is supported by grants through the National Institutes of Health Office of the Director: Regional Medical Centers: 1 OT2 OD026549; 1 OT2 OD026554; 1 OT2 OD026557; 1 OT2 OD026556; 1 OT2 OD026550; 1 OT2 OD 026552; 1 OT2 OD026553; 1 OT2 OD026548; 1 OT2 OD026551; 1 OT2 OD026555; IAA #: AOD 16037; Federally Qualified Health Centers: HHSN 263201600085U; Data and Research Center: 5 U2C OD023196; Biobank: 1 U24 OD023121; The Participant Center: U24 OD023176; Participant Technology Systems Center: 1 U24 OD023163; Communications and Engagement: 3 OT2 OD023205; 3 OT2 OD023206; Community Partners: 1 OT2 OD025277; 3 OT2 OD025315; 1 OT2 OD025337; 1 OT2 OD025276. Pilot Team: OT2 OD023132, OT2 OD023132-02S1, K23HL141447.
Original NIH Protocol Working Group Members: Teri Manolio, Rebekah Rasooly, Josephine Briggs, Rory Collins, Caroline Dilworth, Montserrat Garcia-Closas, Germaine Buck Lewis, Daniel Masys, Jean Olson, Tim Peakman, Bill Riley, Joni Rutter, Paul Sorlie, Elizabeth Wagner, Debbie Winn, Dana Wolff, Kathleen Meister, Luenda Charles, Michael Gaziano, Emily Harris, Carolyn Hutter, Sue Krebs-Smith, Sara Luckhaupt, Steven Moore, Kathleen Merikangas, Dale Sandler, Amy Subar, Jennifer Thornton, Gordon Willis, and Ellen O’Donnell.
Pilot Team Members: Charles Mouton, Katherine Donato, Taneya Koonce, Sheila Kusnoor, Marcia Epelbaum, Mallory Blasingame, Robert Cronin, Rebecca Jerome, Brandy Mapes, Regina Andrade, Rebecca Johnston, David Schlundt, Kemberlee Bonnet, Sunil Kripalani, Kathryn Goggins, Ken Wallston, Mick Couper, and Michael Elliott.
All of Us Participant Provided Committee Members: Paul Harris, Christopher/Chris O’Donnell, Stephanie Fowler, James McClain, Brian Ahmedani, Regina Andrade, Maria Argos, Mona AuYoung, Mark Begale, Bartali Benedetta, Pedro Rey Biel, Louise Bier, Marnie Bloom, Nicolas Borselli, Clinton Brawner, Beth Calhoun, David Cella, Carmen Chinea, David Condon, Rob Cronin, Julien Dedier, Olivier Elemento, Chris Foster, David Garcia, Holly Garriock, John Jackicic, Rebecca Jerome, Daozhong Jin, Christine Johnson, Christine Joseph, Elizabeth Karlson, Mike Kellen, Michelle Kienholz, Andrea LaCroix, Elizabeth Lee, Maria Lopez-Class, Michael Manganiello, Brandy Mapes, Heather Marino, Fernando Martin-Sanchez, Kathy Mazor, Wanda Montalvo, Fatima Munoz, Jyotishman Pathak, Susan Redline, Carolina Rodriguez-Cook, Heather Sansbury, David Schlundt, August Slater, Vicki Smith, Carolina Stamoulous, Susan Tirhi, Rhonda Trousdale, Febe Wallace, Joyce Winkler, Jennifer Worthington, Jennifer Yttriz, and Alvaro Alonso.
Spanish Translation Committee: James McClain, Michael Stokes, Regina Andrade, Oscar Beita, Dianne Beltran, Amaris Castellanos, Carmen Chinea, Rene Covarrubias, Alfredo Ramirez Clark, Alvaro Donayre, Angelica Espinoza, Iliana Faries, Sergio Fernandez-Gonzales, Pablo Gejman, Viridiana Johnson, Maria Lopez-Class, Elizabeth Lugo, Fernando Martin-Sanchez, Rima Matsumoto, Daniel Mompoint, Jorge Navarrete, Vijay Rayanker, Carolina Rodriguez-Cook, Carolina Stamoulous, Gregory Talavera, Solomon Torres, Sujel Valentin, Emma Viera, Carmen Zaldivar, Alejandra Zapien-Hidalgo, Flor McKinley, Ilse Salinas, Janence Ortiz, Janisse Mercado, Jose Guadalupe Martinez Lopez, Marcela Gaitán, and Marcia Lobos.
We also wish to thank All of Us Research Program Direct Eric Dishman as well as our partners as Verily, Vibrent, Scripps, and Leidos.
“Precision Medicine Initiative, PMI, All of Us, the All of Us logo, and The Future of Health Begins with You are service marks of the US Department of Health and Human Services.”
1. MacArthur J, Bowler E, Cerezo M, et al. The new NHGRI-EBI Catalog of published genome-wide association studies (GWAS Catalog). Nucleic Acids Res. 2017;45:D896–D901.
2. Delaney JT, Ramirez AH, Bowton E, et al. Predicting clopidogrel response using DNA samples linked to an electronic health record. Clin Pharmacol Ther. 2012;91:257–263.
3. Schildcrout JS, Denny JC, Bowton E, et al. Optimizing drug outcomes through pharmacogenetics: a case for preemptive genotyping. Clin Pharmacol Ther. 2012;92:235–242.
4. Wilke RA, Ramsey LB, Johnson SG, et al; Clinical Pharmacogenomics Implementation Consortium (CPIC). The clinical pharmacogenomics implementation consortium: CPIC guideline for SLCO1B1 and simvastatin-induced myopathy. Clin Pharmacol Ther. 2012;92:112–117.
5. Mallal S, Phillips E, Carosi G, et al; PREDICT-1 Study Team. HLA-B*5701 screening for hypersensitivity to abacavir. N Engl J Med. 2008;358:568–579.
6. Cavallari LH, Lee CR, Beitelshees AL, et al; IGNITE Network. Multisite investigation of outcomes with implementation of CYP2C19 genotype-guided antiplatelet therapy after percutaneous coronary intervention. JACC Cardiovasc Interv. 2018;11:181–191.
7. Massey JT, Moore TF, Tadros W, Parsons V. Design and estimation for the National Health Interview Survey 1985–94. Vital Health Stat 2. 1989:1–33.
8. Parsons VL, Moriarity CL, Jonas K, Moore TF, Davis KE, Tompkins L. Design and estimation for the National Health Interview Survey, 2006–2015. Vital Health Stat 2. 2014;165:1–53.
9. Sudlow C, Gallacher J, Allen N, et al. UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med. 2015;12:e1001779.
10. Sudlow C, Gallacher J, Allen N, et al. UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med. 2015;12:e1001779.
11. Jensen PB, Jensen LJ, Brunak S. Mining electronic health records: towards better research applications and clinical care. Nat Rev Genet. 2012;13:395–405.
12. Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff (Millwood). 2014;33:1123–1131.
13. Hripcsak G, Albers DJ. Next-generation phenotyping of electronic health records. J Am Med Inform Assoc. 2013;20:117–121.
14. Martin-Sanchez F, Verspoor K. Big data in medicine is driving big changes. Yearb Med Inform. 2014;9:14–20.
15. Baig MM, GholamHosseini H, Moqeem AA, Mirza F, Lindén M. A systematic review of wearable patient monitoring systems—current challenges and opportunities for clinical adoption. J Med Syst. 2017;41:115.
16. Uddin MZ, Khaksar W, Torresen J. Ambient sensors for elderly care and independent living: a survey. Sensors (Basel). 2018;18:2027.
17. Rucco R, Sorriso A, Liparoti M, Ferraioli G, Sorrentino P, Ambrosanio M, Baselice F. Type and location of wearable sensors for monitoring falls during static and dynamic tasks in healthy elderly: a review. Sensors (Basel). 2018;18:pii: E1613.
18. Camomilla V, Bergamini E, Fantozzi S, Vannozzi G. Trends supporting the in-field use of wearable inertial sensors for sport performance evaluation: a systematic review. Sensors (Basel). 2018;18:pii: E873.
19. Johansson D, Malmgren K, Alt Murphy M. Wearable sensors for clinical applications in epilepsy, Parkinson’s disease, and stroke: a mixed-methods systematic review. J Neurol. 2018;265:1740–1752.
20. Papi E, Koh WS, McGregor AH. Wearable technology for spine movement assessment: a systematic review. J Biomech. 2017;64:186–197.
21. Rovini E, Maremmani C, Cavallo F. How wearable sensors can support parkinson’s disease diagnosis and treatment: a systematic review. Front Neurosci. 2017;11:555.
22. Kozitsina AN, Svalova TS, Malysheva NN, Okhokhonin AV, Vidrevich MB, Brainina KZ. Sensors based on bio and biomimetic receptors in medical diagnostic, environment, and food analysis. Biosensors (Basel). 2018;8:pii: E35.
23. Mukhopadhyay SC. Wearable sensors for human activity monitoring: a review. IEEE Sensors J. 2015;15:1321–1330.
24. Hudson K, Lifton R, Patrick-Lake B. The precision medicine
initiative cohort program-building a research foundation for 21st century medicine. Precision Medicine
Initiative (PMI) Working Group Report to the Advisory Committee to the Director, ed. 2015.
26. Investigators AoURP. The All of Us Research Program—Building a Foundation for 21st Century Precision Health. In: Personal Communication Cronin R, ed, 2018.
27. Ford JG, Howerton MW, Lai GY, et al. Barriers to recruiting underrepresented populations to cancer clinical trials: a systematic review. Cancer. 2008;112:228–242.
28. Heller C, Balls-Berry JE, Nery JD, et al. Strategies addressing barriers to clinical trial enrollment of underrepresented populations: a systematic review. Contemp Clin Trials. 2014;39:169–182.
29. National Academy of Sciences NAoE, Institute of Medicine Committee on Underrepresented G, the Expansion of the S, Engineering Workforce P. The National Academies Collection: Reports funded by National Institutes of Health. In: Expanding Underrepresented Minority Participation. 2011.Washington (DC): National Academies Press (US) National Academy of Sciences.
30. Hripcsak G, Forrest CB, Brennan PF, Stead WW. Informatics to support the IOM social and behavioral domains and measures. J Am Med Inform Assoc. 2015;22:921–924.
31. Beatty PC, Willis GB. Research synthesis: the practice of cognitive interviewing. Publ Opin Quart. 2007;71:287–311.
32. DeMuro CJ, Lewis SA, DiBenedetti DB, Price MA, Fehnel SE. Successful implementation of cognitive interviews
in special populations. Expert Rev Pharmacoecon Outcomes Res. 2012;12:181–187.
33. Willis GB. Cognitive Interviewing: A Tool for Improving Questionnaire Design. 2004.Thousand Oaks, California: Sage Publications.
34. Dean E, Head B, Swicegood J. Hill CA, Dean E, Murphy J. Virtual cognitive interviewing using Skype and Second Life. In: Social Media, Sociality, and Survey Research. 2014:Hoboken, NJ: Wiley; 107–132.
35. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377–381.
36. Cronbach L. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297–334.
38. Gaziano JM, Concato J, Brophy M, et al. Million veteran program: a mega-biobank to study genetic influences on health and disease. J Clin Epidemiol. 2016;70:214–223.
39. Cella D, Riley W, Stone A, et al; PROMIS Cooperative Group. The patient-reported outcomes measurement information system (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. J Clin Epidemiol. 2010;63:1179–1194.
40. Wallston KA, Cawthon C, McNaughton CD, Rothman RL, Osborn CY, Kripalani S. Psychometric properties of the brief health literacy screen in clinical practice. J Gen Intern Med. 2014;29:119–126.
41. Haider AH, Schneider EB, Kodadek LM, et al. Emergency department query for patient-centered approaches to sexual orientation and gender identity: the EQUALITY study. JAMA Intern Med. 2017;177:819–828.
42. Parikh-Patel A, Allen M, Wright WE; California Teachers Study Steering Committee. Validation of self-reported cancers in the California Teachers Study. Am J Epidemiol. 2003;157:539–545.
43. Alvarez RA, Vasquez E, Mayorga CC, Feaster DJ, Mitrani VB. Increasing minority research participation through community organization outreach. West J Nurs Res. 2006;28:541–560; discussion 561.
44. Williams MM, Meisel MM, Williams J, Morris JC. An interdisciplinary outreach model of African American recruitment for Alzheimer’s disease research. Gerontologist. 2011;51(suppl 1):S134–S141.
45. Gauthier MA, Clarke WP. Gaining and sustaining minority participation in longitudinal research projects. Alzheimer Dis Assoc Disord. 1999;13(suppl 1):S29–S33.
Cognitive interviews; Cohort studies; Health surveys; Online surveys; Precision medicine; Questionnaires
Supplemental Digital Content
Copyright © 2019 Wolters Kluwer Health, Inc. All rights reserved.