Bahl, Rajiv MD, PhD*; Martines, Jose MD, PhD*; Ali, Nabeela MD, MPH†; Bhan, Maharaj K. MD‡; Carlo, Wally MD§; Chan, Kit Yee PhD¶; Darmstadt, Gary L. MD, MS∥; Hamer, Davidson H. MD**; Lawn, Joy E. MBBS, MRCP, MPH††; McMillan, Douglas D. MD, FRCPC‡‡; Mohan, Pavitra MD, MPH§§; Paul, Vinod MD, PhD¶¶; Tsai, Alexander C. MD, PhD∥∥; Victora, Cesar G. MD***; Weber, Martin W. MD, PhD†††; Zaidi, Anita K. M. MBBS, SM‡‡‡; Rudan, Igor MD, DSc, PhD, MPH§§§¶¶¶
From the *Department of Child and Adolescent Health, World Health Organization, Geneva, Switzerland; †John Snow Inc., Islamabad, Pakistan; ‡Department of Biotechnology, Government of India, New Delhi, India; §Department of Pediatrics, University of Alabama at Birmingham, Birmingham, AL; ¶Nossal Institute for Global Health, The University of Melbourne, Melbourne, Australia; ∥Department of International Health, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD; **Center for International Health and Development and Department of International Health, Boston University School of Public Health, Boston, MA; ††Saving Newborn Lives/Save the Children-US, Capetown, South Africa; ‡‡Department of Pediatrics, Dalhousie University, Halifax, Canada; §§UNICEF India, New Delhi, India; ¶¶All India Institute of Medical Sciences, New Delhi, India; ∥∥Langley Porter Psychiatric Institute, University of California at San Francisco, CA; ***Federal University of Pelotas, Pelotas, Brazil; †††World Health Organization, Jakarta, Indonesia; ‡‡‡Aga Khan University, Karachi, Pakistan; §§§Croatian Centre for Global Health, University of Split Medical School, Split, Croatia; and ¶¶¶Department of Public Health Sciences, the University of Edinburgh Medical School, Scotland, UK.
I.R. received support as a consultant of Child Health and Nutrition Research Initiative (CHNRI) during the conduct of this study. There were no other sources of funding and all coauthors (except I.R.) volunteered their time to conduct this study.
R. B., J. M., and M. W. are staff members of the World Health Organization. The authors alone are responsible for the views expressed in this publication and they do not necessarily represent the decisions or the stated policy of the World Health Organization.
R.B. led and coordinated the process. IR and colleagues designed the CHNRI methodology and adapted it for this process. I.R., J.M., and R.B. planned the process and wrote the first draft of the paper. I.R. computed the intermediate and final scores. A.C.T. and K.Y.C computed agreement statistics scores. All other authors submitted their scores to R.B., reviewed the manuscript and provided important intellectual inputs.
Address for correspondence: Rajiv Bahl, Department of Child and Adolescent Health and Development, World Health Organization, 20 Ave Appia, Geneva, Switzerland. E-mail: firstname.lastname@example.org.
Newborn infections claim an estimated 1.4 million lives each year and remain responsible for approximately one-third of the world's 4.0 million neonatal deaths.1 Neonatal deaths, which now constitute about 40% of deaths in children under 5 years, have until recently remained largely unaddressed as a global health concern.2 To reduce neonatal mortality to half of the year 2000 levels and thus achieve the Millennium Development Goal for child survival by 2015, much more needs to be accomplished, and many challenges must be overcome.1,2
Although effective and simple interventions for prevention and treatment of newborn infections exist, they do not reach the majority of neonates in developing countries.3 This gap between knowledge and practice is due in large part to poor coverage with health services, shortage of health care providers, and issues related to access to referral services. The result is that a large proportion of neonatal infection deaths occur in community settings, frequently at home.
One of the key reasons for inadequate coverage of effective interventions is lack of knowledge on how to implement existing cost-effective interventions at scale in low resource settings.3,4 This gap in knowledge can only be filled by appropriately targeted research.5 Research on newborn care, including prevention and treatment of newborn infections, was ranked the highest among child-health themes in a recent study using the Delphi method.6
The Department of Child and Adolescent Health and Development (CAH) of the World Health Organization (WHO) used the methodology developed by the Child Health and Nutrition Research Initiative (CHNRI) to define research priorities for the major causes of child deaths to focus global research investments.5,7–10 In this article, we present the results for research priorities on newborn infections.
The CHNRI priority-setting methodology has been proposed as a tool for those who develop research policy and/or invest in health research.5 The implementation of CHNRI methodology involves 3 complementary stages: input from investors/policy makers; input from technical experts; and input from other stakeholders.
Input From Investors/Policymakers
CHNRI defined its objective as promoting rational investment of resources to reduce the present burden of neonatal mortality through health research by 2015. The context was defined in this way to assist the efforts and progress toward the Millenuium Development Goal 4 (reducing global child mortality), to which political commitment has been made. Setting investment priorities for research is difficult because of several factors: uncertainty of research outcomes, risk preferences of the investors (eg, high-risk high-profit or low-risk moderate-profit), agreed targets for disease burden reduction, level of urgency and time frame for expected reduction, systems of values of the stakeholders, and many others. All those factors alter investment priorities in different contexts.5–11
In the case of neonatal deaths due to infections, the context was specified as follows:
* Burden of disease and disability of interest: deaths from newborn infections;
* Population of interest: newborns in all developing countries, where 99% of neonatal deaths occur1;
* Target: reduction of neonatal mortality, contributing to reduction of under 5 child mortality by two-thirds by 2015 (UN's Millennium Development Goal 4);
* Level of urgency: high (because the goal is not being achieved);
* Time frame: to achieve detectable reduction in the rate of neonatal mortality by 2015;
We then defined the criteria to identify research options that would be most likely to provide the greatest “returns on investments” (reduction in number of deaths) within the given context. Based on CHNRI's conceptual framework,5,12 5 criteria are typically proposed for discriminating the competing research investment options, and we applied all of them: (1) ability to answer in an ethical way; (2) likelihood of developing or improving interventions that are effective; (3) likelihood that interventions will be deliverable, affordable, and sustainable; (4) maximum potential impact on mortality burden reduction; and (5) predicted impact of research results on equity in the population.
Input From Technical Experts
The coordinator of the project for WHO/CAH (RB) invited a group of 20 international technical experts with interest in newborn infections to participate in the process. Invitations were based upon an established track record of publications in this area and/or experience in leading international programs. The first task of the technical experts was to propose an extensive list of research questions, developed in a systematic way, according to the framework developed by CHNRI. The framework begins with 4 broad “research domains”: (1) epidemiological research, (2) health systems and policy research, (3) research targeted at improving the existing interventions, and (4) research to develop new interventions. These domains were then divided into more specific, “research options,” which correspond to 3- to 5-year research programs. Experts were invited to specify as many of these options as possible. Within each such option, they were asked to list very specific “research questions,” the answers to which, for example might appear in research articles published in journals. The coordinator from CAH collected the input from each of the 20 experts independently by e-mail to produce a consolidated list of research questions. In producing this list, the coordinator limited the overlap between proposed ideas and ensured that the research questions were phrased in a way that would make the expected new knowledge apparent.
The experts were then invited to score all research questions independently, according to the 5 agreed criteria. As this process is time-consuming, each expert chose to evaluate all the research questions on 2 of the 5 criteria on which he/she felt he/she could provide the best informed independent opinion. For each of the research questions, each expert answered 3 questions targeted to assess the likelihood of the proposed research to comply with each of the priority-setting criteria (Appendix 1). This task was completed by 13 of the 20 experts initially contacted. The a priori decision was to consider the process complete when each of the 5 criteria was scored by at least 4 technical experts. The entire process was conducted via e-mail between January 2007 and February 2008. Further methodological details on this part of the priority-setting process are presented elsewhere in great detail.5,7–12
Input From Other Stakeholders
The third step in the CHNRI methodology is to incorporate the opinions and values from a broader group of stakeholders (eg, expected recipients of the research, taxpayers who fund health research, health workers, journalists and media, experts in ethics, law, political science).9 These stakeholders do not have sufficient expertise and knowledge to directly score research questions. To include their opinions and values, CHNRI consultants interviewed the members of the Global research priority-setting network, which is coordinated from the University of Toronto in Canada. Interviewed members were representative of a broad group of contexts and backgrounds, and the details on the composition of this group and rationale of the process was described in great detail elsewhere.10 Each surveyed examinee was asked to rank the 5 criteria by importance and give weights and minimum required scores (“thresholds”) to each criterion. This information was then used to weight experts’ scores, reflecting the values of stakeholders (see later).10,12
Computation of Overall Research Priority Scores and Assessment of Expert Agreement
All the experts answered the questions listed in Appendix 1 by “Yes” (1 point) or “No” (0 points). They were also allowed to declare an informed but undecided answer (0.5 points) or declare themselves insufficiently informed to answer the question (missing input). In this way, the proposed research questions received a score for each of the 5 criteria–5 “intermediate scores.” These scores are defined as the proportion of maximum possible points scored when an answer was given. They represent a direct measure of collective optimism of at least 4 experts scoring each research question independently. Therefore, each of the listed research questions received 5 intermediate scores ranging from 0% to 100%, 1 for each of the criteria. These 5 proportions were then weighted according to the input from the stakeholders. The weights were applied as follows: the intermediate score related to the criterion “maximum potential for disease burden reduction” was given a weight of 1.75; to the criterion “answerability in an ethical way” a weight of 0.96; to the criterion “predicted effect on equity in the population” a weight of 0.91; to the criterion “deliverability, affordability, and sustainability” a weight of 0.89; and to the criterion “potential contribution to effectiveness” a weight of 0.86.10 The overall research priority score (RPS) was then computed as the weighted mean of all 5 intermediate priority scores.
Because of its transparency and independent scoring by the larger group of experts, the CHNRI methodology has the ability to expose the points of greatest agreement and greatest controversy. The agreement is not assessed using κ statistics because the datasets that CHNRI methodology produces (eg, allowing for the existence of missing responses, “undecided” responses and different number of experts scoring different criteria) are not really appropriate for application of the usual κ statistic.13,14 Instead, for each research investment option we reported the average proportion of scorers that agreed on the 15 questions asked. This is computed for each scored research investment option as:
Equation (Uncited)Image Tools
(where q is a question that experts are being asked to evaluate competing research investment options, ranging from 1 to 15; AEA, average expert agreement). For each evaluated research investment option, AEA is informing us, for an average question, what proportion of scorers gave the same most frequent answer (eg, when AEA is about 60%, this means that for an average question related to a specific research investment option, 3 out of 5 scorers gave the most frequent answer).
The expert group process produced a list of 69 questions, 27 of them related to epidemiological research, 28 to health systems and policy research, 6 to research targeted at improving the existing interventions, and 8 to research to develop new interventions (see web table http://links.lww.com/A642). In this list of 69 questions, we intentionally kept some relatively similar proposed research questions separate to check for consistency of our experts’ scoring. Research questions addressing similar issues ended up with very similar priority scores. Before presenting the final results, we therefore, merged 8 questions with another similar one on the list, resulting in 61 ranks in all.
Nine of 10 research questions related to prevention and management of newborn infections received the highest overall scores (Table 1). All of these research questions were related to improved delivery of known interventions and fell under the domain of health policy and systems research. It is interesting to note that 4 of these questions pertained to community-based delivery of interventions, 4 to facility-based interventions and one to both.
Research questions related to developing new interventions, improving existing interventions, and the epidemiology of nosocomial infections were also among the top 15 questions (Table 2). Most of these priorities could be applied at both community and health facility levels.
The 15 research priorities in Tables 1 and 2 focus on evaluating intervention effectiveness (ranks 2, 11, 12, and 13), developing and evaluating approaches to improve newborn care practices (rank 1), increasing intervention coverage through provision of care at the community level (ranks 3, 5, 6, 10, and 15), and improving quality of care at health facilities (ranks 4, 7, 8, 9, and 14).
For the top 10 research investment options (Tables 1 and 2), “average expert agreement” parameter ranged between 67% and 82%, but it was typically higher than 75%, indicating that 3 out of 4 scorers gave the same answer to an average question related to those options. Considerably more controversy was observed over the research investment options from the bottom of the ranking list, where AEA parameter typically ranged between 58% and 65% (ie, an agreement that was only slightly above that expected by chance for some research investment options).
This priority-setting exercise undertaken by CAH using CHNRI methodology resulted in a set of research priorities mainly focused on health policy and systems research. The priorities target better understanding of the barriers to implementation, effectiveness, and optimization of use of available interventions for prevention of newborn infections and their management at the community and health facility levels.
This finding is not surprising because technical experts were asked to define research priorities that could lead to notable improvements in reduction of newborn infection-related mortality by the year 2015. The overall context that required this short time frame favored the prioritization of questions that proposed to assess and confirm the value of existing and available cost-effective interventions. It also highlighted the value of investments in health systems and policy research that proposed to identify key obstacles to delivery, affordability, and sustainability of implementation of those interventions on a larger scale.
The average expert agreement for the identified research investment priorities was typically higher than 75%, indicating that 3 out of 4 scorers gave the same answer to an average question related to those options. This is considerably above the expectation of the level of agreement if they were assigning scores “0” or “1” randomly (which would be under 50%, as “undecided” answer is also allowed). This shows a good overall level of agreement over the proposed priorities among the experts.
The research priorities identified are consistent with the messages from The Lancet Series on Neonatal Survival.1,2 It is well known that effective interventions exist, which can reduce neonatal mortality by up to 70% if they reach all mothers and newborns. The greatest challenge is to increase coverage of these interventions, and the research priorities presented in this article would help in addressing that challenge. The identified priorities are also in general agreement with the research currently supported by WHO/CAH.
Three limitations of the exercise merit consideration. First, the selected expert group may not be representative of all developing country experts. Second, only about two-thirds of the selected experts completed the entire process despite best efforts. Getting the expert group face-to-face to undertake scoring could have increased the proportion of experts that completed the process, but would have compromised the independent opinion of individual experts. Finally, a risk with this approach of priority setting with a short term perspective (until 2015 only) is that this becomes a barometer of what the involved experts are thinking about and doing, rather than a probing rethink of current research priorities. However, the expert group in this exercise did consider research on a wide variety of new interventions and innovative approaches to deliver interventions before deciding on the final priorities.
Within the context of an existing global consensus on expected mortality reduction (UN's Millennium Development Goal 4), to which political commitment has been made, it is important to prioritize health policy and systems research. These results from implementation of the CHNRI methodology imply that more attention should be given to health policy research, health systems research, operations research, and research that addresses political, economic, social, cultural, behavioral, and infrastructure issues involved in addressing mortality due to neonatal infections. This type of research is not always attractive to investors in health research because the results are unlikely to grab the headlines or be published in journals with high impact factors. However, such implementation research is critical for generating the new knowledge needed to achieve high coverage with life-saving interventions.
The applied CHNRI methodology proved to be a helpful process to systematically list a large number of specific research questions and then score them independently. Although decisions will eventually rest with the investors and policy makers, the CHNRI process provided recommendations by a group of technical experts who made the strengths and weaknesses of a large number of research options transparent. The level of agreement on suggested research can also be informative to potential investors. Involvement of the large group of stakeholders and a wider community in setting research investment priorities is a major challenge, because the criteria of relevance to scientists and technical experts may not always be in accordance with the views and values of those who benefit from research investments. However, the key stakeholders do not have the expertise needed to judge specific research investment options on the criteria. This is why the CHNRI method leaves that task to technical experts, while the stakeholders can still have a say on the relative importance of the criteria used and weight them to influence the final outcome.
The final list of priorities emphasizes the evaluation of existing interventions and the development and testing of new delivery approaches for existing interventions. It also highlights the value of research on preventive measures and treatment and on improving intervention delivery at home, community, and health facility levels. While community-based intervention research is receiving some current investment, rigorous evaluation and cost analyses are almost entirely lacking for research on facility-based interventions and quality improvement.
In conclusion, given a short time frame for expected results (2015) and the lack of progress in improving newborn survival despite the existence of effective interventions, it is not surprising that the majority of top ranking research priorities presented in this article are in the domain of health systems and policy research. We believe that implementation of research studies in developing countries on these priorities would greatly contribute to knowledge that would lead to reduction of global neonatal mortality. We urge funding agencies and investigators to support these research priorities to accelerate reduction of neonatal deaths, particularly due to infections.
The authors thank S. Qazi for his comments and suggestions on the draft article.
1. Lawn JE, Cousens S, Zupan J, et al. Four million neonatal deaths: when? Where? Why? Lancet. 2005;365:891–900.
2. Martines J, Paul VK, Bhutta ZA, et al. Neonatal survival: a call for action. Lancet. 2005;365:1189–1197.
3. Darmstadt GL, Bhutta ZA, Cousens S, et al. Evidence-based, cost-effective interventions: how many newborn babies can we save? Lancet. 2005;365:977–988.
4. Knippenberg R, Lawn JE, Darmstadt GL, et al. Systematic scaling up of neonatal care in countries. Lancet. 2005;365:1087–1098.
5. Rudan I, El Arifeen S, Black RE, et al. Childhood pneumonia and diarrhoea: setting our priorities right. Lancet Inf Dis. 2007;7:56–61.
6. Costello A, Filippi V, Kubba T, et al. Research challenges to improve maternal and child survival. Lancet. 2007;369:1240–1243.
7. Tomlinson M, Chopra M, Sanders D, et al. Setting priorities in child health research investments for South Africa. PLoS Med. 2007;4:E259.
8. Chisholm D, Flisher AJ, Lund C, et al. Scale up services for mental disorders: a call for action. Lancet. 2007;370:1241–1252.
9. Rudan I, Gibson J, Kapiriri L, et al. Setting priorities in global child health research investments: assessment of principles and practice. Croat Med J. 2007;48:595–604.
10. Kapiriri L, Tomlinson M, Gibson J, et al. Setting priorities in global child health research investments: addressing the values of the stakeholders. Croat Med J. 2007;48:618–627.
11. Brown KH, Hess SY, Boy E, et al. Setting priorities for zinc-related health research to reduce children's disease burden worldwide: an application of the CHNRI research priority-setting method. Public Health Nutr. 2008;22:1–8.
12. Rudan I, Chopra M, Kapiriri L, et al. Setting priorities in global child health research investments: universal challenges and conceptual framework. Croat Med J. 2008;49:398–408.
13. Byrt T. How good is that agreement? Epidemiology. 1996;7:561.
14. Cicchetti DV, Feinstein AR. High agreement but low Kappa: II. Resolving the paradoxes. J Clin Epidemiol. 1990;43:551–558.
© 2009 Lippincott Williams & Wilkins, Inc.