Secondary Logo

Journal Logo

Development of the Initial Surveys for the All of Us Research Program

Cronin, Robert M.a,b; Jerome, Rebecca N.c; Mapes, Brandyc; Andrade, Reginac; Johnston, Rebeccac; Ayala, Jenniferc; Schlundt, Davidd; Bonnet, Kemberleed; Kripalani, Sunile,f,g; Goggins, Kathryne,f,g; Wallston, Kenneth A.h; Couper, Mick P.i,j; Elliott, Michael R.k,l; Harris, Paulc; Begale, Markm; Munoz, Fatiman; Lopez-Class, Mariao; Cella, Davidp; Condon, Davidp; AuYoung, Monaq; Mazor, Kathleen M.r; Mikita, Steves; Manganiello, Michaelt; Borselli, Nicholast; Fowler, Stephanieo; Rutter, Joni L.o; Denny, Joshua C.a; Karlson, Elizabeth W.u; Ahmedani, Brian K.v; O’Donnell, Christopher J.w,x the Vanderbilt University Medical Center Pilot Team, and the Participant Provided Information Committee

doi: 10.1097/EDE.0000000000001028
Study Design
Free
SDC

Background: The All of Us Research Program is building a national longitudinal cohort and collecting data from multiple information sources (e.g., biospecimens, electronic health records, and mobile/wearable technologies) to advance precision medicine. Participant-provided information, collected via surveys, will complement and augment these information sources. We report the process used to develop and refine the initial three surveys for this program.

Methods: The All of Us survey development process included: (1) prioritization of domains for scientific needs, (2) examination of existing validated instruments, (3) content creation, (4) evaluation and refinement via cognitive interviews and online testing, (5) content review by key stakeholders, and (6) launch in the All of Us electronic participant portal. All content was translated into Spanish.

Results: We conducted cognitive interviews in English and Spanish with 169 participants, and 573 individuals completed online testing. Feedback led to over 40 item content changes. Lessons learned included: (1) validated survey instruments performed well in diverse populations reflective of All of Us; (2) parallel evaluation of multiple languages can ensure optimal survey deployment; (3) recruitment challenges in diverse populations required multiple strategies; and (4) key stakeholders improved integration of surveys into larger Program context.

Conclusions: This efficient, iterative process led to successful testing, refinement, and launch of three All of Us surveys. Reuse of All of Us surveys, available at http://researchallofus.org, may facilitate large consortia targeting diverse populations in English and Spanish to capture participant-provided information to supplement other data, such as genetic, physical measurements, or data from electronic health records.

From the aDepartment of Biomedical Informatics and Internal Medicine, Vanderbilt University Medical Center, Nashville, TN

bDepartment of Pediatrics, Vanderbilt University Medical Center, Nashville, TN

cVanderbilt Institute for Clinical and Translational Research, Vanderbilt University Medical Center, Nashville, TN

dDepartment of Psychology, Vanderbilt University, Nashville, TN

eDepartment of Medicine, Vanderbilt University School of Medicine, Nashville, TN

fCenter for Clinical Quality and Implementation Research, Vanderbilt University Medical Center, Nashville, TN

gCenter for Effective Health Communication, Vanderbilt University Medical Center, Nashville, TN

hInstitute for Medicine and Public Health, Vanderbilt University Medical Center, Nashville, TN

iSurvey Research Center, University of Michigan. Ann Arbor, MI

jJoint Program in Survey Methodology, University of Maryland, College Park, MD

kDepartment of Biostatistics, School of Public Health, University of Michigan, Ann Arbor, MI

lSurvey Research Center, Institute for Social Research, University of Michigan, Ann Arbor, MI

mVibrent Health, Fairfax, VA

nDepartment of Research and Health Promotion, San Ysidro Health, San Diego, CA

oNational Institutes of Health, Office of the Director, Bethesda, MD

pDepartment of Medical Social Sciences, Northwestern University Feinberg School of Medicine, Chicago, IL

qScripps Whittier Diabetes Institute, Scripps Health, San Diego, CA

rMeyers Primary Care Institute, Worcester, MA

sSpinal Muscular Atrophy Foundation, New York, NY

tHCM Strategists, Washington, DC

uDivision of Rheumatology, Allergy, and Immunology, Department of Medicine, Section of Clinical Sciences, Brigham and Women’s Hospital, Boston, MA

vCenter for Health Policy & Health Services Research, Henry Ford Health System, Detroit, MI

wCardiology Section, Department of Medicine, Veterans Affairs Boston Healthcare System, Boston, MA

xCardiovascular Medicine Division, Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA.

Submitted November 21, 2018; accepted April 2, 2019.

This work was funded by the National Institutes of Health (https://allofus.nih.gov/) OT2OD023132 (R.M.C., R.N.J., B.M.M., R.A., R.J., J.A., D.G.S., K.R.B., S.K., K.G., K.A.W., M.P.C., M.R.E., P.A.H., and J.C.D.), U2COD023196 (J.C.D.), R01AG0532641U24OD023163-01 (M.J.B.), HHSM500201200008I/HHSN263201800208U (F.A.M.), U2CCA186878 (D.C.), U24 OD023176 (M.A.), 3OTOD 024610-01S1 (K.M.M.), OT2OD023206 (M.M. and N.B.), 1OT2OD026553 (E.W.K.), and OT2OD026550 (B.K.A.). Additional support included the National Heart, Blood, and Lung Institute (https://www.nhlbi.nih.gov/) K23HL141447 (R.M.C.), United States Veteran’s Administration Healthcare System (C.J.O.), the National Institute on Aging (https://www.nia.nih.gov/) U2C AG060426-01 (D.M.C.). Program staff of the funding organization, National Institutes of Health, co-contributed to this work through conceptualization, methodology, validation, project administration and review and editing of this manuscript.

The authors report no conflicts of interest.

Supplemental digital content is available through direct URL citations in the HTML and PDF versions of this article (www.epidem.com).

Data availability: All relevant data are within the manuscript and its Supporting Information files or are available from http://researchallofus.org.

Correspondence: Robert M. Cronin, Department of Biomedical Informatics, Vanderbilt University Medical Center, 2525 West End Blvd., Suite 1475, Nashville, TN 37232. E-mail: robert.cronin@vanderbilt.edu.

Precision medicine is an approach to identifying risk factors, etiology, treatment, and prevention of disease emphasizing variability in an individual’s genes, environment, and lifestyle. Precision medicine research has yielded numerous discoveries regarding genomic influences on diseases and drug responses,1–4 which has resulted in improved patient outcomes.5,6 These studies have historically leveraged phenotypic information from physiologic measurements, health surveys, and bioassays. Surveys have been an important part of observational research for decades. Prominent studies such as the National Health and Nutrition Examination Survey,7 the National Health Interview Survey,8 and UK Biobank,9,10 have added substantially to biomedical knowledge in part through survey-based capture of exposures and outcomes. Research has also demonstrated the power of routine clinical information extracted from the electronic health record (EHR); others are also exploring the utility of newer modalities such as wearable biosensors and environmental data.11–23 Collectively, these strategies hold promise for more precise ways to identify patterns in large datasets and, ultimately, understanding of factors that contribute to health and disease.

The All of Us Research Program (All of Us) has the goal of enrolling a longitudinal cohort of at least one million participants reflecting the rich diversity of the United States population,24–26 prioritizing groups historically underrepresented in biomedical research.27–29 A broad range of data will be collected to describe each participant, including surveys, EHRs, biospecimens, physical exams, wearable technologies, and geospatial and environmental sources. These data will enable creation of a robust research resource to facilitate deep exploration of biologic, clinical, social, environmental, and behavioral determinants of health and disease.

All of Us will use surveys to collect data directly from a diverse cohort of participants across the socioeconomic spectrum with different racial/ethnic backgrounds and for whom English may not be their primary language. These surveys are primarily designed to complement information collected from other sources such as EHRs. EHRs do not routinely collect data on many social, environmental, or behavioral determinants of health, or these variables may be difficult to find within the text of EHR clinical notes,30 thus the additional data from surveys have the potential to enable researchers to test hypotheses at a greater scope and level of precision than before.

Surveys also offer an opportunity to engage participants, explicitly recognizing the importance of the information that individuals can share over time about their own sociodemographic characteristics, health, and other factors. In All of Us, participants answer survey questions via an online participant portal, available on both computers and mobile devices, which guide participants through surveys to be completed, although keeping track of their progress and engaging them throughout the life of the Program. This article describes the rigorous process for rapid development, testing, and refinement of the initial three surveys included in the launch of All of Us in 2018.

Back to Top | Article Outline

METHODS

The All of Us survey development process comprised a multidisciplinary collaboration and included: (1) preparatory work such as choosing the scientific domains of focus and examining existing validated instruments to create content; (2) content testing and refinement; and (3) stakeholder agreement on final versions for program deployment (Figures 1 and 2).

FIGURE 1

FIGURE 1

FIGURE 2

FIGURE 2

Back to Top | Article Outline

Drafting Initial Surveys

This process began in October 2015 with an NIH All of Us Protocol Working Group convened by the National Institutes of Health (NIH) Office of the Director, consisting of approximately 25 NIH staff and non-NIH scientists from a variety of disciplines. The working group drafted a list of desired survey domains to be included in All of Us. Priorities of the domains were determined based on a set of criteria (eTable 1; http://links.lww.com/EDE/B518). This group identified and reviewed existing validated survey instruments for use in the All of Us Research Program from October 2015 to March 2016, including 19 instruments from large research endeavors (Table 1). Instrument questions were chosen based on validation evidence in diverse populations, usage in other national studies, and access to their use (lack of copyright).

TABLE 1

TABLE 1

The All of Us Pilot team was formed in early 2016 to continue survey development. The Pilot team employed qualitative methods to test and refine survey content to ensure optimal implementation, described below. This interdisciplinary team consisted of approximately 20 experts representing a range of fields: (1) cognitive sciences and electronic/nonelectronic survey design and related methodology; (2) health literacy and effective health communication; (3) engagement and inclusion of underrepresented minority populations, such as African Americans and Hispanics/Latinos, Lesbian, Gay, Bisexual, Transgender, Queer or Questioning, and Intersex, and rural residents; (4) information science, for evidence review and synthesis; (5) racially and ethnically diverse bilingual research staff proficient in English and Spanish; and (6) biomedical informatics, for guidance around data collection methods, data harmonization, and quality and quantity of information.

The Pilot team built upon the materials originally drafted by the Working Group. The three surveys prioritized by the Program for initial testing and included as part of the national launch of All of Us recruitment, guided by the information most typically collected at time of enrollment in large cohort studies included: The Basics (sociodemographic questions), Overall Health (general health overview), and Lifestyle (tobacco, alcohol, and drug questions).

The team created a standardized process, described below, to test each survey’s readability, response format and options, participants’ confidence in their ability to answer the questions accurately, and other key issues, such as redundancy and sensitivity.

As part of this phase of work, the Pilot team also collated all source instruments from which content was drawn to create a metadata database. The metadata database was used to ensure that principles of survey development were adhered to, including (1) maintenance of scientific validity of the original survey instruments, (2) conversion of interviewer-led surveys to an online participant-completed format, and (3) usage of validated items widely used in the same diverse populations recruited to the Program. The Pilot team reviewed the metadata for gaps related to these guiding principles.

As the Program grew, survey creation became the responsibility of the Participant-Provided Information Committee. This committee is a group of approximately 10–15 experts tasked with overseeing the entire survey development cycle, comprised of the types of domain experts mentioned above, as well as representatives from participant recruitment sites. This committee has been instrumental in refining content as well as determining new areas of focus. Table 2 describes prioritized domains and current development status.

TABLE 2

TABLE 2

Back to Top | Article Outline

Testing Sample

The Pilot team developed an approach for qualitative testing, including cognitive interviewing and web-based testing. Participants were recruited from a robust and diverse pool of ~5000 individuals who had previously expressed interest in helping to develop All of Us content, as well as through additional methods such as in-person events with community partners. These recruitment approaches enabled the Pilot team to reach minority populations and those with limited access to the internet, as well as rural and urban areas.

Back to Top | Article Outline

Qualitative and Quantitative Testing

Cognitive interviewing employed a “think aloud” approach, exploring participants’ understanding about each question and response option. Probes, including interviewer prompts and follow-up questions, were used as needed to elicit detailed discussion of each item.31–33 Cognitive interviews were conducted in person or by Skype31,33,34 based on participant preference. Transcripts of interview audio recordings were analyzed qualitatively. A hierarchical coding system was developed to thematically assess the qualitative data. Quotes supporting the different codes were recorded as part of the analysis. Interviews continued until saturation was achieved.

Online qualitative and quantitative testing, done in parallel with cognitive interviews, intended to augment the data from cognitive interviews by engaging a larger number of diverse participants. Online testing employed the Research Electronic Data Capture35 system hosted in a secure online environment, which required that each participant login using unique account credentials to help ensure only the proper participant completed the survey. The online qualitative testing allowed participants to first answer the question, then share feedback about the clarity, understandability, and sensitivity of each question and its response options (Figure 3); in quantitative testing, used to estimate the time to complete each survey, participants only answered questions without a feedback option. Where appropriate, analysis of these results included exploratory factor analysis using principal components extraction and varimax rotation, frequency statistics, and assessment of internal consistency using Cronbach’s alpha.36

FIGURE 3

FIGURE 3

Back to Top | Article Outline

Translation and Testing in Spanish

The surveys were developed and tested in both English and Spanish (Figures 1 and 2). Before testing, the English surveys were translated into Spanish. If the English language questions had an equivalent Spanish version from the original source instrument, then that Spanish equivalent was used as the primary translated version. However, if the English version did not have an existing Spanish translation, those questions went through a multistep translation process developed by the Pilot team. This included primary translation by a third-party company (MotionPoint, Coconut Creek, FL), during which a certified professional translator translated the content into Spanish, followed by secondary review of the translation by community reviewers to ensure the content was easily understandable to a wide range of Spanish speakers with diverse educational attainment and country of origin. This secondary review also identified opportunities for improving readability.

For the first Spanish language iteration of the initial surveys (The Basics, Overall Health, and Lifestyle), secondary translation review was conducted by a small group of Program stakeholders representing diverse geographical native Spanish speakers ranging from Mexico, Spain, Venezuela, and El Salvador to ensure that the Spanish language review captured wide ethnogeographic and cultural differences. This group also reviewed the final recommendations for the translated Spanish materials used in the Program.

Testing of the Spanish-language materials primarily included recruiting diverse Spanish-speaking participants across the United States at in-person events and connecting with community organizations embedded within Latino communities. The approach for identifying any critical changes in the Spanish-language materials followed the same process outlined for the English versions.

Back to Top | Article Outline

Review by Key Program Stakeholders

The Pilot team reviewed and proposed further refinements to the materials based on review of the original validated instruments and analyses from the cognitive interviews and online testing. The survey materials and their accompanying proposed list of changes were then reviewed by key Program leadership, including the Participant-Provided Information and Steering Committees. Feedback from these groups was used to inform a refined iteration of each survey, leading to a second round of testing (Figures 1 and 2 and eTable 2; http://links.lww.com/EDE/B518). After this second round of testing, the Pilot team provided recommendations that led to a final review and approval by key Program leadership.

Back to Top | Article Outline

Implementation in Participant Portal

In the final step of development, the three surveys were incorporated into the All of Us participant portal, the mechanism for survey self-administration in the program. These surveys are available to participants once they sign into the portal, allowing for completion in their own time.

Back to Top | Article Outline

RESULTS

Process Overview

The testing and refinement process successfully implemented for the first three modules in the program included two rounds of cognitive interviews and online testing, complemented by expert review and input after each round of testing. The results of each of the development phases are described in further detail below.

Back to Top | Article Outline

First Round of Testing and Revisions

The first round of testing focused on both qualitative and quantitative testing of the English survey materials. Cognitive interviews were conducted with 74 participants. A total of 337 individuals provided online feedback on the English surveys, of which 225 provided qualitative feedback, and 112 provided quantitative feedback (Table 3). Qualitative testing revealed major themes that demonstrated issues with clarity, understandability, and sensitivity (eTable 3; http://links.lww.com/EDE/B518). A summary of the estimated completion time data calculated based on this round of testing is included in Table 4.

TABLE 3

TABLE 3

TABLE 4

TABLE 4

Back to Top | Article Outline

Second Round of Testing and Revisions

The second round of cognitive interviewing focused on expanding participant diversity and testing both English (n = 48) and Spanish (n = 47) versions (total n = 95). Online testing in English was completed with 236 participants (qualitative testing n = 159, quantitative testing n = 77) (Table 3). There were no new substantial changes in items that emerged from online qualitative testing in the second round. See Table 4 for estimated completion times. Review of the data led to very minor recommendations involving changing wording and adding clarifying language to some of the questions.

Back to Top | Article Outline

The Yield of this Development Process: Final Revisions to Survey Content

A summary of the over 40 recommended changes based on English language testing is included in Table 5. Minor changes consisted of small edits in phrasing of a question or response options (e.g., converting questions that were originally administered by interviewers in the parent survey to a format appropriate for self-administration). Major changes included more substantive modifications to a question or response options when noteworthy concerns were expressed by participants during qualitative testing. For example, the questions about race, ethnicity, and gender that were included in the first round of qualitative testing were modified and retested in the second round of testing based on input from participants and key Program stakeholders. These modifications led to refinement of our approach to race and ethnicity. We ultimately leveraged extensive testing completed by the US Census Bureau, which found that a combined question for capturing participant-reported race and ethnicity was the strongest approach for gathering complete and accurate data.37 The recommendations for changes in the Spanish language version included minor changes that implemented more simplified and common terminology. In addition, testing Spanish-language materials resulted in feedback and recommendations that went beyond translation, and included conceptual changes to questions to make them more understandable and accessible to their specific communities.

TABLE 5

TABLE 5

We explored concerns from members of the Program, such as about asking potentially sensitive questions regarding topics like gender identity and sexual orientation. Our testing did not confirm anticipated concerns but instead found an appreciation of being asked these questions among participants, including those self-identifying as sexual and gender minorities.

All questions were finalized by the committees and are summarized with their original instruments in eDocuments 1–6; http://links.lww.com/EDE/B518. The summary of iterative changes for both English and Spanish are summarized in eTable 4; http://links.lww.com/EDE/B518.

Back to Top | Article Outline

Exploratory Factor Analysis

Although much of the data in the three surveys were not amenable to further statistical analysis, an exploratory factor analysis was appropriate for two subcomponents (i.e., PROMIS, Brief Health Literacy Screen) of the English version of Overall Health. This factor analysis showed three distinct factors with coefficients alpha being 0.92 for General Physical Health, 0.81 for Emotional Health, and 0.55 for Health Literacy (eTable 5; http://links.lww.com/EDE/B518).

Back to Top | Article Outline

DISCUSSION

We created an iterative process leveraging diverse experts to develop and refine materials for collection of participant-provided information for All of Us that is applicable to a diverse audience, leverages existing validated surveys, and supports English and Spanish. We initially launched three surveys and are following this model for ongoing development of future surveys in All of Us.

Back to Top | Article Outline

Lessons Learned

Other large consortia such as the Million Veterans Program38 and the UK Biobank9,10 have included survey materials as a core data component; however, approaches for combining survey items from multiple sources in the context of a large research program are sparse. We learned several lessons useful for future Program development, as well as others undertaking similar work. First, we discovered generally minor issues with clarity and sensitivity for some module questions drawn from previously validated survey instruments. Although questions from certain validated instruments, such as the PROMIS Global Health Scale39 and the Brief Health Literacy Screen,40 performed well in our testing (eTable 3; http://links.lww.com/EDE/B518), we implemented minor modifications for other items (eTable 4; http://links.lww.com/EDE/B518). Modifications included adding examples to clarify a question (e.g., providing the number of cigarettes in a pack); new response options (e.g., e-cigarettes); and explanatory text before some items. As many of the module questions are derived from existing national surveys, we wanted to avoid substantial wording revisions, as this would compromise established scientific validity. Instead, we targeted all newly created explanatory text to the fifth-grade level and focused on identifying areas within the question text where minor revisions could lead to major improvement in readability. Second, we explored potential issues of concern to All of Us. In fact, similar to other findings,41 participants expressed appreciation for the Program’s recognition of the importance of asking about gender identity and sexual orientation, which initially concerned members of the Program. Third, we successfully leveraged prior extensive participant testing within national programs, such as the US Census,37 as our findings echoed the Census Bureau’s observations that a combined approach for querying race and ethnicity is more aligned with the way participants identify themselves and, thus, allows gathering more granular data. Fourth, we found that many validated questions from studies that may have been developed for a specific population, such as the California Teachers Study,42 only required minor wording changes, but otherwise performed well in a more diverse population. Finally, collaborating with consortium experts led to substantial improvements in integrating survey materials into the larger context of the Program. This collaboration led to a volume of survey items reasonable to ask participants to complete at enrollment, within the larger scope of enrollment activities such as consent and physical exam.

Our experience emphasizes the value of systematically vetting multiple languages to ensure optimum survey deployment, as well as the importance of testing with various dialects of a specific language. Creating and assessing surveys in both English and Spanish led to improved clarity, although maintaining concordance between translations. Testing in both languages ensured alignment in understandability and accessibility for the English and Spanish versions. This experience continues to inform the consortium’s work regarding the complexities of testing surveys in different languages. The translation process included review by members from multiple Spanish speaking regions to develop surveys understood by Spanish speakers from different regions. Instituting multiple versions of Spanish surveys is worth consideration; however, we did not find evidence within our interviews with participants that indicated this need.

We learned multiple lessons about cognitive interviewing strategies in this population. First, this testing process yielded valuable lessons learned regarding strategies for overcoming challenges in recruiting underrepresented or hard to reach populations. Over time, the Pilot team expanded recruitment to include methods such as in-person recruitment within the community, facilitating increased enrollment of harder-to-reach populations such as those without internet access, those with lower educational attainment, racial/ethnic minority populations, and Spanish-speaking participants. This face-to-face community interaction was an effective method to recruit certain populations that were more likely to engage in research opportunities through a trusted and familiar entity such as an established community organization. As echoed by others these methods require more time, but are critical to ensure inclusion of diverse populations.43–45 Second, cognitive interview probes authored by content experts allowed a deeper understanding of what needed to be explored to ensure accurate answers. Third, using a web application to perform interviews over the internet provided a cost-effective way to interview people who were not located locally. Fourth, our sample size was large, and we achieved saturation before reaching our full sample size. Because enrollment of underrepresented populations is a major aim of the All of Us Research Program, we wanted to be thorough in our attempts to include key populations in this component of the program planning. Future efforts of this type will likely include smaller enrollment targets. However, based on our experiences, we feel that the size and demographic characteristics of the sample are important and should be tailored to best fit the scope and goals of the project. For example, researchers interested in exploring readability and comprehension alone may reach saturation with a small sample. In instances where sensitivity and individual perspectives are also being sought, researchers may want to consider a larger and more diverse sample size.

Back to Top | Article Outline

Limitations

We acknowledge several limitations: (1) Our testing did not include all populations underrepresented in biomedical research. Although we included very important populations, including sexual and gender minorities, those of low socioeconomic status, and Spanish-only speakers, gaps may require additional testing with other populations. (2) The speed at which initial survey development work proceeded limited our ability initially to include individuals from certain key groups, such as lower educational attainment. However, this was addressed during online testing and out second round of cognitive interviews. (3) The validity of combining questions from existing instruments was not thoroughly tested. To mitigate this, we intentionally minimized changes from existing instruments and executed entire scales or sets of questions about a topic area from a single instrument. (4) E-mail communication was the main method of contact for study recruitment, potentially limiting our reach of those less comfortable with technology. (5) Small sample size and potential lack of representativeness limits generalization of our exploratory factor analysis. (6) Online surveying poses an inherent risk, however small, of receiving a response from someone other than the intended participant. Finally, (7) we tested primarily on a computer platform and not on different technology platforms such as smartphones. Our plan will be to test future surveys on different platforms.

Back to Top | Article Outline

Deployment and Future Directions

To achieve the ambitious scope and scale of All of Us, surveys will need to be modular to not overwhelm participants, accessible on a digital platform so surveys can be completed in a variety of settings, available to be completed at a participant’s own pace, and engaging, so participants continue to contribute after initial enrollment in the Program.

Building upon this successful process for development of these initial materials, All of Us is developing other surveys (Table 2). Future areas of interest for the program will include rollout of these surveys, engaging participants to complete these additional surveys, and repeat administration of some surveys to reflect participant changes over time. As the Program evolves, survey development will continue to be a core activity. As described above, the marriage of scientific value, engagement, and participant experience will remain an important consideration for future work in this area. Future testing and integration with other sources of information, such as EHR data, mobile sensors, and a range of technology platforms, genetics, and physical measurements, will help enhance the value and completeness of All of Us data for future hypothesis exploration. Further validation of our findings related to the surveys in the launch of the Program will be needed to ensure what we found in our initial testing holds true for the larger cohort. Translation into other languages and the testing of these translations will also be an important area of activity as the Program grows.

Participant-provided information is a critically important part of the data that will power All of Us. The survey questions and response options must be carefully documented and communicated to researchers in a way that makes these data accessible and easy to integrate into external aspects of clinical research. An All of Us survey codebook is under development and is being designed to map All of Us survey items to standard vocabularies that can help align these data with EHR data where possible (e.g., for diagnoses). Finally, All of Us surveys reflect a mixture of questions from validated instruments in the public domain, accompanied by supplementary; questions that are not currently publicly available. All of Us will publish these surveys and their metadata, as they become available, at http://researchallofus.org.

Back to Top | Article Outline

CONCLUSIONS

This flexible process combined multidisciplinary expertise with Program leadership input and proven methods to create and refine surveys that are appropriate for use in the diverse participant population of All of Us. The process, which was generalizable across multiple survey domains, formed a firm roadmap for the development and testing of future materials. Other large consortia that target a diverse population in multiple languages could employ this process to create surveys that supplement other data sources, such as genetics and EHR data.

Prioritization and generation of All of Us surveys will continue in close collaboration with representatives across the NIH Institutes and Centers to ensure item integrity and scientific validity and help ensure that questions reflect the mission of the NIH to improve health outcomes broadly. Other surveys may be driven by participant interest. Participant engagement and experience will also be important components of future development. Gathering and incorporating participant-provided information in a systematic way will enhance the scientific validity and breadth of information obtained from All of Us, leading to exciting new advances in the era of precision medicine.

Back to Top | Article Outline

RESEARCH ETHICS AND INFORMED CONSENT

The research assessments described in this manuscript were approved by the Institutional Review Board of the All of Us Research Program. Those participants that completed cognitive interview gave oral consent to do so.

Back to Top | Article Outline

ACKNOWLEDGMENTS

We wish to thank our participants who have joined All of Us and contributed to participant-provided information; helped refine early materials; engaged in the development and evaluation of the surveys, and provided other ongoing feedback. We thank the countless other coinvestigators and staff across all awardees and partners without which All of Us would not have achieved our current goals. All of Us is supported by grants through the National Institutes of Health Office of the Director: Regional Medical Centers: 1 OT2 OD026549; 1 OT2 OD026554; 1 OT2 OD026557; 1 OT2 OD026556; 1 OT2 OD026550; 1 OT2 OD 026552; 1 OT2 OD026553; 1 OT2 OD026548; 1 OT2 OD026551; 1 OT2 OD026555; IAA #: AOD 16037; Federally Qualified Health Centers: HHSN 263201600085U; Data and Research Center: 5 U2C OD023196; Biobank: 1 U24 OD023121; The Participant Center: U24 OD023176; Participant Technology Systems Center: 1 U24 OD023163; Communications and Engagement: 3 OT2 OD023205; 3 OT2 OD023206; Community Partners: 1 OT2 OD025277; 3 OT2 OD025315; 1 OT2 OD025337; 1 OT2 OD025276. Pilot Team: OT2 OD023132, OT2 OD023132-02S1, K23HL141447.

Original NIH Protocol Working Group Members: Teri Manolio, Rebekah Rasooly, Josephine Briggs, Rory Collins, Caroline Dilworth, Montserrat Garcia-Closas, Germaine Buck Lewis, Daniel Masys, Jean Olson, Tim Peakman, Bill Riley, Joni Rutter, Paul Sorlie, Elizabeth Wagner, Debbie Winn, Dana Wolff, Kathleen Meister, Luenda Charles, Michael Gaziano, Emily Harris, Carolyn Hutter, Sue Krebs-Smith, Sara Luckhaupt, Steven Moore, Kathleen Merikangas, Dale Sandler, Amy Subar, Jennifer Thornton, Gordon Willis, and Ellen O’Donnell.

Pilot Team Members: Charles Mouton, Katherine Donato, Taneya Koonce, Sheila Kusnoor, Marcia Epelbaum, Mallory Blasingame, Robert Cronin, Rebecca Jerome, Brandy Mapes, Regina Andrade, Rebecca Johnston, David Schlundt, Kemberlee Bonnet, Sunil Kripalani, Kathryn Goggins, Ken Wallston, Mick Couper, and Michael Elliott.

All of Us Participant Provided Committee Members: Paul Harris, Christopher/Chris O’Donnell, Stephanie Fowler, James McClain, Brian Ahmedani, Regina Andrade, Maria Argos, Mona AuYoung, Mark Begale, Bartali Benedetta, Pedro Rey Biel, Louise Bier, Marnie Bloom, Nicolas Borselli, Clinton Brawner, Beth Calhoun, David Cella, Carmen Chinea, David Condon, Rob Cronin, Julien Dedier, Olivier Elemento, Chris Foster, David Garcia, Holly Garriock, John Jackicic, Rebecca Jerome, Daozhong Jin, Christine Johnson, Christine Joseph, Elizabeth Karlson, Mike Kellen, Michelle Kienholz, Andrea LaCroix, Elizabeth Lee, Maria Lopez-Class, Michael Manganiello, Brandy Mapes, Heather Marino, Fernando Martin-Sanchez, Kathy Mazor, Wanda Montalvo, Fatima Munoz, Jyotishman Pathak, Susan Redline, Carolina Rodriguez-Cook, Heather Sansbury, David Schlundt, August Slater, Vicki Smith, Carolina Stamoulous, Susan Tirhi, Rhonda Trousdale, Febe Wallace, Joyce Winkler, Jennifer Worthington, Jennifer Yttriz, and Alvaro Alonso.

Spanish Translation Committee: James McClain, Michael Stokes, Regina Andrade, Oscar Beita, Dianne Beltran, Amaris Castellanos, Carmen Chinea, Rene Covarrubias, Alfredo Ramirez Clark, Alvaro Donayre, Angelica Espinoza, Iliana Faries, Sergio Fernandez-Gonzales, Pablo Gejman, Viridiana Johnson, Maria Lopez-Class, Elizabeth Lugo, Fernando Martin-Sanchez, Rima Matsumoto, Daniel Mompoint, Jorge Navarrete, Vijay Rayanker, Carolina Rodriguez-Cook, Carolina Stamoulous, Gregory Talavera, Solomon Torres, Sujel Valentin, Emma Viera, Carmen Zaldivar, Alejandra Zapien-Hidalgo, Flor McKinley, Ilse Salinas, Janence Ortiz, Janisse Mercado, Jose Guadalupe Martinez Lopez, Marcela Gaitán, and Marcia Lobos.

We also wish to thank All of Us Research Program Direct Eric Dishman as well as our partners as Verily, Vibrent, Scripps, and Leidos.

Precision Medicine Initiative, PMI, All of Us, the All of Us logo, and The Future of Health Begins with You are service marks of the US Department of Health and Human Services.”

Back to Top | Article Outline

REFERENCES

1. MacArthur J, Bowler E, Cerezo M, et al. The new NHGRI-EBI Catalog of published genome-wide association studies (GWAS Catalog). Nucleic Acids Res. 2017;45:D896–D901.
2. Delaney JT, Ramirez AH, Bowton E, et al. Predicting clopidogrel response using DNA samples linked to an electronic health record. Clin Pharmacol Ther. 2012;91:257–263.
3. Schildcrout JS, Denny JC, Bowton E, et al. Optimizing drug outcomes through pharmacogenetics: a case for preemptive genotyping. Clin Pharmacol Ther. 2012;92:235–242.
4. Wilke RA, Ramsey LB, Johnson SG, et al; Clinical Pharmacogenomics Implementation Consortium (CPIC). The clinical pharmacogenomics implementation consortium: CPIC guideline for SLCO1B1 and simvastatin-induced myopathy. Clin Pharmacol Ther. 2012;92:112–117.
5. Mallal S, Phillips E, Carosi G, et al; PREDICT-1 Study Team. HLA-B*5701 screening for hypersensitivity to abacavir. N Engl J Med. 2008;358:568–579.
6. Cavallari LH, Lee CR, Beitelshees AL, et al; IGNITE Network. Multisite investigation of outcomes with implementation of CYP2C19 genotype-guided antiplatelet therapy after percutaneous coronary intervention. JACC Cardiovasc Interv. 2018;11:181–191.
7. Massey JT, Moore TF, Tadros W, Parsons V. Design and estimation for the National Health Interview Survey 1985–94. Vital Health Stat 2. 1989:1–33.
8. Parsons VL, Moriarity CL, Jonas K, Moore TF, Davis KE, Tompkins L. Design and estimation for the National Health Interview Survey, 2006–2015. Vital Health Stat 2. 2014;165:1–53.
9. Sudlow C, Gallacher J, Allen N, et al. UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med. 2015;12:e1001779.
10. Sudlow C, Gallacher J, Allen N, et al. UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med. 2015;12:e1001779.
11. Jensen PB, Jensen LJ, Brunak S. Mining electronic health records: towards better research applications and clinical care. Nat Rev Genet. 2012;13:395–405.
12. Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff (Millwood). 2014;33:1123–1131.
13. Hripcsak G, Albers DJ. Next-generation phenotyping of electronic health records. J Am Med Inform Assoc. 2013;20:117–121.
14. Martin-Sanchez F, Verspoor K. Big data in medicine is driving big changes. Yearb Med Inform. 2014;9:14–20.
15. Baig MM, GholamHosseini H, Moqeem AA, Mirza F, Lindén M. A systematic review of wearable patient monitoring systems—current challenges and opportunities for clinical adoption. J Med Syst. 2017;41:115.
16. Uddin MZ, Khaksar W, Torresen J. Ambient sensors for elderly care and independent living: a survey. Sensors (Basel). 2018;18:2027.
17. Rucco R, Sorriso A, Liparoti M, Ferraioli G, Sorrentino P, Ambrosanio M, Baselice F. Type and location of wearable sensors for monitoring falls during static and dynamic tasks in healthy elderly: a review. Sensors (Basel). 2018;18:pii: E1613.
18. Camomilla V, Bergamini E, Fantozzi S, Vannozzi G. Trends supporting the in-field use of wearable inertial sensors for sport performance evaluation: a systematic review. Sensors (Basel). 2018;18:pii: E873.
19. Johansson D, Malmgren K, Alt Murphy M. Wearable sensors for clinical applications in epilepsy, Parkinson’s disease, and stroke: a mixed-methods systematic review. J Neurol. 2018;265:1740–1752.
20. Papi E, Koh WS, McGregor AH. Wearable technology for spine movement assessment: a systematic review. J Biomech. 2017;64:186–197.
21. Rovini E, Maremmani C, Cavallo F. How wearable sensors can support parkinson’s disease diagnosis and treatment: a systematic review. Front Neurosci. 2017;11:555.
22. Kozitsina AN, Svalova TS, Malysheva NN, Okhokhonin AV, Vidrevich MB, Brainina KZ. Sensors based on bio and biomimetic receptors in medical diagnostic, environment, and food analysis. Biosensors (Basel). 2018;8:pii: E35.
23. Mukhopadhyay SC. Wearable sensors for human activity monitoring: a review. IEEE Sensors J. 2015;15:1321–1330.
24. Hudson K, Lifton R, Patrick-Lake B. The precision medicine initiative cohort program-building a research foundation for 21st century medicine. Precision Medicine Initiative (PMI) Working Group Report to the Advisory Committee to the Director, ed. 2015.
25. Health NIo. Program Overview—All of Us|National Institutes of Health. Available at: https://www.ncbi.nlm.nih.gov/pubmed/. Accessed 25 December 2017.
26. Investigators AoURP. The All of Us Research Program—Building a Foundation for 21st Century Precision Health. In: Personal Communication Cronin R, ed, 2018.
27. Ford JG, Howerton MW, Lai GY, et al. Barriers to recruiting underrepresented populations to cancer clinical trials: a systematic review. Cancer. 2008;112:228–242.
28. Heller C, Balls-Berry JE, Nery JD, et al. Strategies addressing barriers to clinical trial enrollment of underrepresented populations: a systematic review. Contemp Clin Trials. 2014;39:169–182.
29. National Academy of Sciences NAoE, Institute of Medicine Committee on Underrepresented G, the Expansion of the S, Engineering Workforce P. The National Academies Collection: Reports funded by National Institutes of Health. In: Expanding Underrepresented Minority Participation. 2011.Washington (DC): National Academies Press (US) National Academy of Sciences.
30. Hripcsak G, Forrest CB, Brennan PF, Stead WW. Informatics to support the IOM social and behavioral domains and measures. J Am Med Inform Assoc. 2015;22:921–924.
31. Beatty PC, Willis GB. Research synthesis: the practice of cognitive interviewing. Publ Opin Quart. 2007;71:287–311.
32. DeMuro CJ, Lewis SA, DiBenedetti DB, Price MA, Fehnel SE. Successful implementation of cognitive interviews in special populations. Expert Rev Pharmacoecon Outcomes Res. 2012;12:181–187.
33. Willis GB. Cognitive Interviewing: A Tool for Improving Questionnaire Design. 2004.Thousand Oaks, California: Sage Publications.
34. Dean E, Head B, Swicegood J. Hill CA, Dean E, Murphy J. Virtual cognitive interviewing using Skype and Second Life. In: Social Media, Sociality, and Survey Research. 2014:Hoboken, NJ: Wiley; 107–132.
35. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377–381.
36. Cronbach L. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297–334.
37. Jones NA. Update on the US Census Bureau’s race and ethnic research for the 2020 Census. 2017. Available at: https://www.census.gov/content/dam/Census/newsroom/press-kits/2014/article_race_ethnic_research_2020census_jones.pdf. Accessed 01 December 2017.
38. Gaziano JM, Concato J, Brophy M, et al. Million veteran program: a mega-biobank to study genetic influences on health and disease. J Clin Epidemiol. 2016;70:214–223.
39. Cella D, Riley W, Stone A, et al; PROMIS Cooperative Group. The patient-reported outcomes measurement information system (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. J Clin Epidemiol. 2010;63:1179–1194.
40. Wallston KA, Cawthon C, McNaughton CD, Rothman RL, Osborn CY, Kripalani S. Psychometric properties of the brief health literacy screen in clinical practice. J Gen Intern Med. 2014;29:119–126.
41. Haider AH, Schneider EB, Kodadek LM, et al. Emergency department query for patient-centered approaches to sexual orientation and gender identity: the EQUALITY study. JAMA Intern Med. 2017;177:819–828.
42. Parikh-Patel A, Allen M, Wright WE; California Teachers Study Steering Committee. Validation of self-reported cancers in the California Teachers Study. Am J Epidemiol. 2003;157:539–545.
43. Alvarez RA, Vasquez E, Mayorga CC, Feaster DJ, Mitrani VB. Increasing minority research participation through community organization outreach. West J Nurs Res. 2006;28:541–560; discussion 561.
44. Williams MM, Meisel MM, Williams J, Morris JC. An interdisciplinary outreach model of African American recruitment for Alzheimer’s disease research. Gerontologist. 2011;51(suppl 1):S134–S141.
45. Gauthier MA, Clarke WP. Gaining and sustaining minority participation in longitudinal research projects. Alzheimer Dis Assoc Disord. 1999;13(suppl 1):S29–S33.
Keywords:

Cognitive interviews; Cohort studies; Health surveys; Online surveys; Precision medicine; Questionnaires

Supplemental Digital Content

Back to Top | Article Outline
Copyright © 2019 Wolters Kluwer Health, Inc. All rights reserved.