Phase 2: Submission of Priority Research Questions
The first online survey, using Inquisium by Cvent, was shared using multiple dissemination pathways. The aim was to ensure inclusion of a broad range of stakeholders including researchers, program managers, clinicians, implementers, community partners, advocates, and young people, covering different epidemic and geographic settings. Using targeted snowballing, concentrated dissemination efforts were undertaken through (1) WHO country offices to ensure engagement of national program managers, (2) established community and youth networks, (3) postings on WHO and IAS websites and social media, and (4) the IAS and CIPHER member list serve of 3631 professionals working on pediatric and adolescent HIV. Respondents were encouraged to share the survey with their relevant networks.
Participants could submit up to 10 priority research questions across the 3 research areas, based on their knowledge and expertise for either one or both populations. Respondents were also asked to tag their question with the relevant research area and domain. The first online survey and associated communications were available in English, French, Spanish, and Portuguese.
Phase 3: Thematic Content Analysis of Priority Research Questions Submitted
Non-English responses were back-translated. The data were cleaned by removing incomplete questions and those that did not meet the scope of the exercise (eg, relating to prevention, adult populations etc.). The research questions were then screened and sorted into population groups and research area of testing, treatment, and service delivery.
Qualitative thematic content analysis was used to analyze the research questions submitted.14,15 After the data were cleaned and sorted, recurring patterns were identified and categorized into themes and subthemes by A.A. and C.I. with the additional review and technical support from M.P. and M.V. The themes and subthemes identified were compared, harmonized, and refined across both populations. Where less than 3 questions were identified for any given theme, these were excluded from subsequent steps. Based on the thematic analysis, the research questions that addressed similar concepts were merged and formulated to find optimal balance between the breadth and details of the research questions submitted. This helped to minimize the overall number of questions to be scored in phase 4. The final collated lists of research questions were reviewed by the expert working group for clarity, consistency, and structure.
Phase 4: Scoring of Research Questions
The collated merged research questions were uploaded onto SurveyMonkey Inc (Location: San Mateo, CA; Main Website: www.surveymonkey.com) to form the second survey. SurveyMonkey allowed the order of questions to be randomized per respondent, thus reducing preferential bias due to scoring fatigue. Respondents of survey 1 were invited to score a minimum of 2 research areas that related most to their expertise for one or both populations using predetermined criteria (Table 1). The CHNRI working group indicates that collective opinion reaches saturation at around 45–55 experts; therefore, survey 2 was closed once at least 45 individuals had responded per research area.16 Survey 2 was only available in English as the non-English response rate of survey 1 was below the predefined threshold of 10%.
Based on other CHNRI processes, participants scored the answers to each criterion, which were then converted to a score (“yes” scored 100, “possibly” scored 50, and “no” scored 0). Participants could leave a response blank if they did not feel sufficiently informed to make a judgment. Blank responses were left out of the calculation of both numerator and denominator.8 Rankings were based on the total Research Priority Score (RPS) according to the formula:
RPS = [(answerability × 0.86) + (impact × 1.56) + (implementation × 0.77) + (equity × 0.81)]/4. The RPS was based on the mean suggested weights according to published guidelines from CHNRI stakeholders8,17 but rescaled to 4 (instead of 5) criteria with a maximum total numerator of 4, then adjusted to a 100-point scale. In addition, the Average Expert Agreement (AEA) scores were calculated, which represent the average proportion of scorers who agreed on responses for each of the 4 criteria asked (Fig. 1).
Phase 5: Identification of Top Five Thematic Priorities per Research Areas
This final phase was an addition to the established CHNRI method described in phases 1–4 above. The working group members, with additional experts chosen to increase diversity (geographical and age) and strengthen some key areas of expertise, were charged with identifying 5 priority themes among the top 10 ranked questions in each research area. This involved (1) reviewing the ranked lists in the context of existing policy, systematic reviews, recently published research, and planned and ongoing research, and (2) participation in a webinar with structured facilitated discussion to reach consensus.
To inform this phase, an Excel template was generated per population, and divided into testing, treatment, and service delivery. Each area was organized by (1) the CHNRI results of ranked research questions, (2) the relevant WHO HIV recommendations18,19 and related systematic reviews, and (3) corresponding ongoing and planned studies. A broad PubMed search identified publications from June 2015, when latest systematic reviews were conducted for WHO HIV guidelines, to February 2017. Broad search terms relating to the 2 populations and HIV were used. These were coded thematically according to research areas and themes and provided as an additional resource. Ongoing and planned research was informed by the participants of survey 1 who, in addition to submitting the research questions, provided information on research projects being undertaking or planned. All types of study designs were accepted. This was strengthened with results from additional searches in clinicaltrials.gov and sorted to match the top 10 ranked questions per research area.
Two virtual meetings per population were held with the expert group to review the templates and make decisions on the top 5 research themes. The webinar was structured into 4 consensuses building steps20 (1) gathering initial feedback and clarifications, (2) facilitated discussion, (3) defining emerging proposals and changes, and (4) testing for consensus on the final top 5 themes. All participants had the opportunity to review and provide final agreement on the research themes.
No identifiable information was linked to respondent submissions and no IP addresses were recorded during the process. Contact information was used to forward respondents the second survey and to send out reminders. All participants could exit both surveys at any time.
A total of 375 participants submitted 1735 priority research questions (an average of 4.6 questions per person) between September and October of 2016, with 58% submitting questions on both adolescent and pediatric HIV. Submissions were received from over 70 countries from all WHO regions, with most from the African region. Of the participants who submitted research questions, 55% (n = 206) identified themselves as researchers. The characteristics of the respondents for surveys 1 and 2 show similar composition (Table 3).
After analysis, the final collated lists included 51 research questions and 61 research questions related to pediatric HIV and adolescent HIV, respectively (Fig. 2). Of the participants of survey 1 who were invited to participate in survey 2, 48% (n = 181) scored the collated lists of research questions.
Efforts to gather information on ongoing and planned studies resulted in the identification of the following: 37 testing, 57 treatment, and 19 for service delivery studies for pediatric HIV and 27 testing, 49 treatment, and 30 service delivery studies for adolescent HIV, mapped against the top 10 research questions per area. The PubMed scoping resulted in 249 and 65 published studies corresponding to the top 10 research questions for pediatric HIV and adolescent HIV, respectively. The final top themes identified and the full CHNRI ranked lists of research questions are discussed in separate publications.11,12
The CHNRI methodology proved to be a highly comprehensive process that ensured a systematic, transparent, open, and inclusive approach to setting research priorities in pediatric and adolescent HIV. The novel adaptation of the CHNRI process for this exercise involved developing 5 top-priority thematic areas from the top 10 ranked research questions, within the context of the current research landscape. This process was also distinctive for 3 reasons. First, it addressed 2 populations in parallel, both children and adolescents. As we see children are surviving into adolescence there is an increased overlap in expertise, reflected by 58% of respondents who submitted questions for both populations. Second, the collaboration of WHO and CIPHER in leading the process, joined by their mandates to guide research, allowed for extensive reach to an existing group of engaged experts and IAS members involved in the HIV response. This was further facilitated by the already strongly connected and collaborative nature of HIV pediatric and adolescent networks and support of the expert working group. Third, it covered the HIV cascade of care across testing, treatment, and service delivery.
Limitations and Strengths
The Top Five Thematic Areas
The additional phase to identify top themes served several functions: to ensure the relevance of the product, to provide a clear digestible outcome for users, to acknowledge the context of current research efforts, and to both facilitate communication and action. This phase was also in response to rapidly evolving research, especially because over the past 2 years, there has been an increased focus across several international agendas and initiatives12,13,21,22 recognizing the unique needs and services required for these populations. The strength of this additional step was that it outlined the existing body of evidence, highlighted key evidence gaps, and provided up-to-date knowledge of the research landscape, complementing that of the expert working group. Themes were used to separate this final step from the ranked lists of research questions identified. Although one previous study conducted content analysis to identify themes across 2 separate CHNRI exercises,23 to the best of our knowledge, this is the first example of identifying prioritized thematic areas based on the current evidence base. It is important to note that it was neither possible to comment on the quality of the ongoing and planned research nor the recently published literature identified, nor was it feasible to account for studies recently completed but not yet published. With the provision of the extensive mapping and the systematically formatted online consultations, it was the role of the expert resource group to highlight additional evidence gaps and reach consensus on the thematic outcomes. Although the themes communicate clearly the priority areas, the full ranked lists of research questions identified through the CHNRI method are also available and should be considered for future research.11,12
It was important that there was wide-ranging multisectorial and multidisciplinary stakeholder inclusion to ensure that priorities reflected the range of priority areas across the HIV cascade and to support ownership of the exercise.24 Extensive outreach was performed, including snowballing, with the disadvantage that for the first survey, a respondent rate could not be calculated. Nonresponse rate bias therefore cannot be disregarded for survey 1. However, overall, there was broad geographical representativeness, including a majority from the African region, which is reflective of the geographic center of the HIV epidemic. Despite active efforts, engagement of younger age groups, ministries of health, and community-based organizations, who are an important part of the HIV response, was lower and future methods of how best to increase their participation will need to be considered.
Analysis of Questions Submitted
Efforts were made to ensure that the length of the research lists for survey 2 were manageable to reduce scoring fatigue and promote response rate, while at the same time addressing both the detail and breadth of the questions submitted by stakeholders. It cannot be discounted that this approach may have introduced bias5 from those compiling the final lists. However, the focused health topic area probably allowed for more extensive merging of complementing research ideas compared with other CHNRI processes that often address multiple health issues. Language was kept as consistent as possible with the submitted questions and, where appropriate, in line with language used by WHO HIV guidelines. Because the number of questions submitted was around 5 questions per person, future CHNRI efforts could propose that 5 or fewer questions be submitted to reduce time and resources required for analysis phase.
Impact of the Process
Assessing the impact of research prioritization efforts is important, although challenging. One group assessed the impact of their work by quantifying the volume of publications since their prioritization exercise. Despite performing a comprehensive review, they concluded that the outcomes could not clearly be attributable to the priorities set.25 Future efforts could consider repeating the policy and research mapping exercise in 5–10 years and comparing it with the results of the mapping performed for this process. This may give an indication of the research volume in the priority areas identified. CHNRI has also suggested conducting interviews with key stakeholders to evaluate their uptake and implementation of the research agenda.5 Given the wide stakeholder engagement in this process and to address the question of attributability, the impact of this process will be in part measured through a survey of the different stakeholder groups on their implementation of the agendas, including funders and researchers. Through this process, CHNRI proved to be an inclusive, clear, transparent, and systematic method for setting research priorities, thus improving on previous consultative methods used by the WHO Department of HIV and IAS.
To date, this is the largest example of the CHNRI priority setting exercise, in terms of stakeholders reached, in pediatric and adolescent HIV, and the first to incorporate top thematic areas based on current evidence. Led by global organizations and experts working in HIV, the resulting research agendas were developed through extensive inclusion of global stakeholders. The overall collective knowledge of the 375 individuals who participated is likely to be more reflective of the priorities in the field than previous research prioritization efforts performed. The additional step of reviewing this collective knowledge in the context of an up-to-date research landscape and highlighting thematic priorities further focuses the research agendas on what is really needed. The research agendas are intended to be adopted by funders and researchers to increase funding available and focus the research where evidence is most needed. A call to action has been launched by CIPHER and WHO urging all stakeholders to support implementation of the agendas through engagement in their respective roles, and highlighting the importance of involving people living with HIV in all aspects of its implementation in a meaningful and responsible way. The impact it will have on improving outcomes for these populations will depend on strong stakeholder leadership, optimized research efforts, and political and financial commitment.
The authors thank the 375 experts who actively participated in the exercise by submitting and those who also scored questions. Special thanks and acknowledgement for the time commitment of the expert working group for providing guidance and inputs throughout this research prioritization process (Elaine Abrams, Columbia University & International Center for AIDS Care and Treatment Program; Rachel Baggaley, World Health Organization; Georgina Caswell, International HIV/AIDS Alliance; Lucie Cluver, University of Oxford & University of Cape Town; Shaffiq Essajee, World Health Organization; Jane Ferguson, Africa Health Research Institute & London School of Hygiene & Tropical Medicine; Laura Guay, Elizabeth Glaser Pediatric AIDS Foundation; Rohan Hazra, Eunice Kennedy Shriver National Institute of Child Health; Dorothy Mbori-Ngacha, UNICEF; Angela Mushavi, Ministry of Health Zimbabwe; Carlo Oliveras, Adolescent HIV Treatment Coalition; David Ross, World Health Organization; Annette Sohn, American Foundation for AIDS Research & TREAT Asia; and Aditi Sharma, independent consultant, International Advocacy Strategist & Campaigner).
The authors also thank the expert resource group for their advice and technical inputs into defining the final top priority themes (Moherndran Archary, University of KwaZulu Nata; Pim Brouwers, National Institutes of Health (NIH); Diana Gibb, MRC Clinical Trials Unit; Grace John-Stewart, University of Washington; Sabrina Kitaka, Makerere University College of Heath Sciences; Valerian Leroy, Université Paul Sabatier; Chewe Luo, UNICEF; Lynne Mofenson, Elizabeth Glaser Pediatric AIDS Foundation; Paul Nary, UNICEF ESARO; Thanyawee Puthanakit, Chulalongkorn University; Lara Vojnov, World Health Organization; Lucy Wanjiku, International Community of Women living with HIV; and Nicola Willis, Africaid).
The authors also thank NIH/AHISA for their contributions to the ongoing studies mapping exercise (Alicia Livinski, Rachel Sturke, and Susan Vorkoper).
2. Penazzato M, Amzel A, Abrams EJ, et al. Pediatric treatment scale-up: the unfinished agenda of the global plan. J Acquir Immune Defic Syndr. 2017;1:S59–S65.
3. UNAIDS. On the Fast-Track to End AIDS
. 2016–2021 Strategy. Geneva, Switzerland: UNAIDS; 2015.
4. Yoshida S. Approaches, tools and methods used for setting priorities in health research in the 21st century. J Glob Health. 2016;6:010507.
5. Rudan I, Yoshida S, Chan KY, et al. Setting health research priorities
using the CHNRI method: VII. A review of the first 50 applications of the CHNRI method. J Glob Health. 2017;7:011004.
6. Rudan I, Gibson JL, Ameratunga S, et al. Setting priorities in global child health research investments: guidelines for implementation of the CHNRI method. Croat Med J. 2008;49:720–733.
7. Hindin MJ, Sigurdson Christiansen C, Ferguson J. Setting research priorities
for adolescent sexual and reproductive health in low and middle income countries. Bull World Health Organ. 2013;91:10–18.
8. Nagata JM, Ferguson BJ, Ross DA. Research priorities
for eight areas of adolescent health in low- and middle-income countries. J Adolesc Health. 2016;59:50–60.
9. Rollins N, Chanza H, Chimbwandira F, et al. Prioritizing the PMTCT implementation research agenda
in 3 African countries: INtegrating and scaling up PMTCT through implementation REsearch (INSPIRE). J Acquir Immune Defic Syndr. 2014;67:S108–S113.
11. Penazzato M, Vicari M, Irvine C, et al. A global research agenda
for paediatric HIV. J Acquir Immune Defic Syndr. 2018;78(suppl 1):S10–S15.
12. Armstrong A, Nagata J, Baggely R, et al. A global research agenda
for adolescent HIV
. J Acquir Immune Defic Syndr. 2018;78(suppl 1):S16–S21.
13. Rudan I. Setting health research priorities
using the CHNRI method: IV. Key conceptual advances. J Glob Health. 2016;6:010501.
14. Javadi M, Zarea K. Understanding thematic analysis and its pitfall. J Clie Care. 2016;1:34–40.
15. Elo S, Kääriäinen M, Kanste O, et al. Qualitative content analysis: a focus on trustworthiness. SAGE Open. 2014;4:1–10.
16. Yoshida S, Rudan I, Cousens S. Setting health research priorities
using the CHNRI method: VI. Quantitative properties of human collective opinion. J Glob Health. 2016;6:010503.
17. Kapiriri L, Tomlinson M, Chopra M, et al. Setting priorities in global child health research investments: addressing values of stakeholders. Croat Med J. 2007;48:618–627.
18. World Health Organization. Consolidated Guidelines on the Use of Antiretroviral Drugs for Treating and Preventing HIV Infection: Recommendations for a Public Health Approach. Geneva, Switzerland: World Health Organization; 2016. Available at: https://www.ncbi.nlm.nih.gov/books/NBK374294/
. Accessed August 28, 2017.
20. Seeds for Change. A Consensus Handbook, Cooperative Decision Making for Activists, Coops and Communities. United Kingdom: Seeds for Change, Lancaster Co-operative Ltd; 2013. Available at: https://www.seedsforchange.org.uk/consensus
. Accessed August 28, 2017.
23. Nagata JM, Hathi S, Ferguson BJ, et al. Research priorities
for adolescent health in low- and middle-income countries: a mixed-methods synthesis of two separate exercises. J Glob Health. 2018;8:010501.
24. Viergever RF, Olifson S, Ghaffar A, et al. A checklist for health research priority setting: nine common themes of good practice. Health Res Policy Syst. 2010;8:36.
25. Odone A, Matteelli A, Chiesa V, et al. Assessing the impact of defining a global priority research agenda
to address HIV-associated tuberculosis. Trop Med Int Health. 2016;21:1420–1427.
Keywords:Copyright © 2018 Wolters Kluwer Health, Inc. All rights reserved.
Child Health and Nutrition Research Initiative (CHNRI); research priorities; adolescent HIV; pediatric HIV; AIDS; research agenda