Hyder, Adnan A. MD, MPH, PhD; Harrison, Rachel A.; Kass, Nancy ScD; Maman, Suzanne MHS, PhD
In the past few years, ethical debates about cross-cultural research have appeared frequently in the literature.1–3 In turn, these debates have sparked a growing interest in international research ethics and have inspired collaborative efforts in addressing international ethical concerns. Many were predominantly concerned about the growing amount of research that is being sponsored by wealthy countries and conducted in resource-poor countries.4 Some of the concerns raised focused on potential exploitation of research participants or poor countries, whereas others focused on appropriate standards of care and participants' posttrial access to interventions.1,5–7 These and related challenges in international research ethics demand a complex set of responses, including responses from the host countries themselves. To that end, various deliberations and reports on international research ethics in the last decade have consistently advocated the development of local research-ethics expertise in low-middle-income countries.8
Controversies in collaborative research may be particularly acute in Africa—a resource-poor, very-low-income, developing region, and one further devastated by the enormity of the HIV/AIDS epidemic.9,10 Indeed, clinical trials related to HIV have precipitated ethics debates throughout the international medical community; yet the debates could be more effectively addressed if there were an increased information exchange about these issues between the developed and developing countries.11 Research systems that are already in place in Africa have been described as “doing things for or to the poor, [but policies should] start strengthening the capacities of the poor to do things themselves.”12 Lack of education in ethics, lack of resources, lack of trained personnel, and attempts to borrow principles of ethical review from developed countries are some of the challenges faced by national health research systems in Africa. These deficiencies are compounded by “brain drain” (the migration of African professionals to the developed world), representing a net intellectual loss to the region.13
The Fogarty International Center (FIC) of the National Institutes of Health (NIH) recognized the need for capacity development in research ethics in developing countries and launched the International Bioethics Education and Career Development Award in 2000. The objectives set forth by FIC for this award are to improve the quality of international ethics training, to support advanced training of professionals in developing country, and to develop intensive short courses for individuals involved in the ethical review of human participant research.14
The Johns Hopkins Fogarty African Research Ethics Training Program (JHF)15 received one of five bioethics awards granted in the initial funding cycle by the FIC. This grant provided financial support for the program for an initial period of four years and then, through a competitive renewal, support was obtained for another four years. The program brings African researchers tothe United States to receive training in research ethics, enabling them to apply and spread these concepts in their home countries. In this article, we present a case study describing the JHF program's attempt to help strengthen research ethics capacity in Africa. Our overarching aim in presenting this case study is to identify critical elements of an effective research ethics capacity development process. More specifically, the objectives of our article are to describe the JHF program, to assess the outputs of the JHF training program during four years, and to analyze the implications of the current model of the JHF training program for further growth of this and similar FIC programs. It is not our goal to examine whether the Johns Hopkins program is in any way better or worse than other bioethics training efforts; arguably, a variety of efforts may be important to satisfy heterogeneous needs and regions. Nonetheless, it is likely that an examination of one program's achievements, strengths, and weaknesses can help improve capacity-development efforts in bioethics in similar programs and regions in the future.
Each year of the program, we select three trainees and three candidates to put on a wait list from the pool of applications we receive from candidates in sub-Saharan Africa. The majority of trainees in the JHF program engage in a one-year fellowship. Trainees spend the first six months at the Johns Hopkins Bloomberg School of Public Health and the Johns Hopkins Berman Bioethics Institute in the United States. During this time, trainees complete bioethics, research ethics, and research methodology courses, they observe IRB meetings from more than one institution, and they attend ethics and research seminars and journal clubs. Trainees travel regularly to the NIH to participate in ethics seminars at the NIH department of clinical bioethics. Critically, significant attention is devoted in these first six months to one-on-one mentoring toward the completion of an extensive proposal for a practicum that is completed during the last six months of training. Trainees then spend the next six months of the training program in their home countries, conducting a funded, independent practicum project related to research ethics.
In this article, we assess the JHF training program using various materials produced during the first four years of its existence. We extracted information from the following sources: curriculum and training materials; application records; informal progress notes and evaluation forms submitted by trainees to their advisors; transcripts from the trainees' coursework; resumes; and formal progress reports submitted by trainees.
The framework we chose to assess the program is based on a systems approach, which explores four aspects of the program at three different time periods (Chart 1). In a generic systems approach, the four levels of assessment are inputs, processes, outputs, and outcomes. The three designated time periods are pretraining, training, and posttraining. Inputs are the elements needed to carry out specific processes, and they include teaching materials, faculty, institutional resources, and trainees. Classes, mentoring, and the practicum are examples of program processes, which are the dynamic interaction of inputs for a specified purpose. The next level of evaluation is the output, or immediate results of processes; these are products from the program, such as the trainees themselves and their research publications, reports, and presentations. Finally, the outcomes are assessed, establishing the goals eventually to be achieved by a program. The anticipated overall outcome of this type of initiative is to contribute toward a positive change in research ethics capacity in Africa.
To implement an assessment of the JHF program using the systems approach, we compiled data from the aforementioned materials into three databases corresponding to three time periods: pre-, intra-, and posttraining. All trainees were asked to review the information contained in the databases for accuracy and comprehensiveness and to provide permission for use of their information. Each of the databases represented information for one of the three separate time periods, so the subsequent evaluation required us to apply the systems approach to each time period.
The goal of the JHF training program is to provide sustained, in-depth research ethics training appropriate for scientists from sub-Saharan Africa. Our intent is to focus on relevant capacity development and not to instill a Western style of research ethics in these countries. To operationalize the goal, four core competencies of education in research ethics were developed at the program's outset: theory and principles, teaching of research ethics, empirical research in research ethics, and ethics review boards. Each trainee is exposed to all four of the competencies, but they each choose one as a concentration about which they do extensive reading and in which they will develop their practicum project. The theory and principles competency focuses on the conceptual underpinnings of research ethics. Teaching research ethics is devoted to developing materials for effectively teaching research ethics in the trainee's home country. The empirical research competency focuses on identifying topics, methods, and opportunities for the conduct of empirical research in research ethics. Finally, the ethics review board competency focuses on the elements necessary for the establishment of a board and a quality review, and what capacity development is required to achieve these.
To achieve these competencies, program instructors adopted a variety of teaching approaches: classroom teaching, a weekly seminar just for the trainees in the program, intensive mentoring, research training, IRB observation, and a required practicum (Table 1). Interactive classroom teaching is the format of the three core courses at Johns Hopkins University (JHU) Bloomberg School of Public Health, which provide a foundational knowledge of research ethics for the trainees. The course titled Ethical Issues in Public Health aims to introduce students to the principles and theory of bioethics and to apply these to case studies in public health and health policy (Table 1). Research Ethics and Integrity discusses ethics principles and theories as they apply to specific cases in research ethics. The third formal required course is Ethics of Public Health Practice in Developing Countries, which encourages a systematic consideration of the ethical responsibility and roles of all parties involved in health care delivery and health research in the developing world.
Concurrently with these courses, the trainees also participate in three seminar courses designed specifically for the training program. One is a weekly seminar attended by the trainees, JHU faculty, and the program coordinator. This seminar alternately has sessions focusing on a topic illustrated by relevant articles (to be presented by trainees), presentations by trainees on their practicum, presentations by trainees of original research ethics cases, and guest speakers (Table 1). In seminars, for example, we have discussed justice concerns and the place of incentives in research in resource-poor areas. Moreover, because this course is the weekly venue where all program faculty and participants convene, it is an opportunity to address any pending logistical or administrative matters. The second seminar is a tutorial in research methodology and study design. This seminar is premised on the assumption that one cannot engage in sophisticated discussions of research ethics without a foundational competency in research methods. This seminar also was developed for these trainees and is flexible and responsive to whatever background knowledge in research methodology current trainees bring. The third seminar is on qualitative and social science research methodology. We developed this course in response to the interest of several trainees in conducting qualitative research for their practica and their need for more guidance on how to conduct it. The three seminars ensure the use of proper methodology and are particularly helpful in that trainees are required to develop their own projects that often involve research, to be implemented in their practica. Their research skill development is also enhanced by the guidance and input of their mentors.
Along with the coursework, the seminars offer many experiences that present opportunities for reinforcing and applying the content of the required courses. The trainees are required to observe regular meetings of one IRB and are afforded the opportunity to attend many others, given that seven different IRBs meet regularly on the Johns Hopkins medical campus, and another 17 meet at the NIH. Trainees are provided with all IRB proposals to review in advance of the meeting so that they can follow the deliberations. Those trainees who observe multiple IRBs are able to compare and contrast different approaches to review, deliberation, and documentation (Table 1). The trainees also have the opportunity to attend a variety of university seminars, including those that are sponsored by the Berman Bioethics Institute, the department of international health, and the department of health policy and management at Johns Hopkins, and the department of clinical bioethics at the NIH. These seminars include journal clubs, guest lectures, and discussion groups. JHF program leaders also invite two to three external program faculty each year, most of whom are from Africa, to come to JHU and meet with trainees collectively and individually.
Throughout their six months in residence at JHU, trainees meet regularly with their primary JHU mentor (a member of the JHF executive committee), particularly to develop their practica. Throughout their stay, they are also required to meet individually with the other executive committee members to obtain different perspectives for practicum development and unique professional advice. In general, some of the most important professional development in the program happens during this regular mentoring.
The arrangement of the JHF faculty is graduated in terms of involvement and responsibility. The principal contributors of time and effort are the members of the executive committee, which is responsible for the general success and coordination of the program. The executive committee consists of three faculty members from the Johns Hopkins Bloomberg School of Public Health. These members come from the departments of health policy and management and international health, with interests in different backgrounds ranging from health systems, to bioethics, to social science. This diversity among executive committee members enriches each aspect of the program's process, from applicant admission to the practicum project, bringing both multidisciplinary strengths and the ability to respond to trainee interests.
The associate faculty consists of professionals from different departments at JHU, the NIH, associated research ethics programs, and African professionals of relevant fields. These associates can serve in a range of roles from professor, to mentor, to networking resource, all of which are crucial to the development of the trainees through their continued success and involvement in research ethics. The associated faculty were selected for their content knowledge, expertise in research in Africa, or a specific bioethics issue. For example, a trainee in year three of the program was matched with an associate faculty member working on HIV/AIDS in Africa, whereas another trainee in year four was matched with another faculty member on the basis of a common interest in malaria research in Zambia. In addition, associate faculty members are invited to JHU as guests to interact with trainees. During these visits, the guest faculty member will give a lecture and meet with each trainee individually to assist specifically in his or her practicum development and general bioethics professional development.
The practicum is the juncture in the program at which the trainees implement all they have learned in the classroom into a practical application that will contribute to research ethics capacity in their home country. It also allows for a mentored period of engaging in research ethics work, an area that may have been less prominent in the professional lives of these trainees previously. There is mentoring and funding for the practicum, so the trainees can “practice” being professional research ethicists with the support of program faculty.
As we briefly mentioned in the Case Study section above, we employed a generic systems approach to program assessment, examining four levels of assessment (inputs, processes, outputs, and outcomes) during three designated time periods (pretraining, training, and posttraining).
An analysis of the training program begins with the pretraining time period. The inputs of this stage comprise the advertisements (emails, listserves, word of mouth) used to disseminate information about the program and to solicit applications. Applications are received via mail, fax, and e-mail, and have increased in number during the four years (Figure 1). We evaluate applicants on the basis of many different criteria: previous academic performance, previous experience with research, prior experience in research ethics/research ethics committees (REC), demonstration of future commitment to ethics, institutional support, and a statement of objectives for research training. Each of these factors is rated on a scale from 1 to 5, with one being the lowest rating. Executive committee members who review the applications complete an evaluation sheet for each candidate and provide a recommended action of high admit, admit, uncertain, or reject. It is also possible for the applicant to be recommended for a short summer course in research ethics or for a two-year masters program. The final output of the review is the selection of three trainees and three wait-listed applicants.
The outcome of the pretraining segment is the trainees' six-month residence at JHU. The outcome is dependent on the applicants' acceptance of the offer to enroll in the training program, acceptance of the conditions of admission, success in obtaining a visa, scheduling of flights, and ultimately following through with all of these arrangements to arrive in time for the program's commencement. In four years, only one admitted trainee could not arrive at JHU because of visa problems, but this trainee deferred participation until the subsequent year, when he successfully obtained a visa in time for participation. Table 2 summarizes the key characteristics of the 12 trainees as reported by the trainees themselves. Overall, 75% of the JHF trainees were male, and their professions ranged from university faculty and staff, to physicians, to various government positions, all with reasonable levels of experience and excellent educational backgrounds. On entry, two had bachelor degrees, three had master degrees, and seven had doctorates. All were selected because of their clear commitment to research ethics and their track records of being accomplished in the endeavors they had tackled thus far. They also displayed a diverse set of specific interests in the health sector, such as nutrition or health research information systems. As a result, trainees included a philosophy professor from West Africa, an REC administrator of a new REC, a pediatrician who works with orphans, and a young staff member of an HIV vaccine group.
The trainees' successful arrival at JHU marks the beginning of the training period, in which courses and seminars, IRB involvement, and the creation and completion of the practicum are all considered inputs to training. These inputs are incorporated into the training processes of teaching, mentoring, and networking. Each trainee engages in a productive relationship with a mentor, who is assigned to the trainee by the JHF executive committee on the basis of common professional interests, and creates valuable connections through networking with associate faculty who can assist the trainee in his or her area of interest. Table 3 summarizes the areas of concentration selected by JHF trainees and the nature of the 12 practica that have been completed to date. The popularity of ethics review and empirical research as areas of concentration is evident, and the dual nature of most practica is also clear, reflecting the multiple and practical objectives set by these trainees for themselves.
Ideally, the culmination of these processes leads to the desired output of the trainees' more extensive knowledge about bioethics and research methodology, refined skills in critical reading of literature, and thoughtful consideration of research questions and ethics inquiries. The projected outcome of the training is a professional with a core background in all four competencies and significant expertise in one of the four, who adds to a critical mass of trained ethics professionals in Africa.
The posttraining period is the time in which the trainee is expected to contribute to the mission of research ethics capacity development in Africa. The inputs into the posttraining period are initiated by the trainee, by the faculty, or by others. More extensive commitments that require the trainee's initiative, such as his or her interaction with the ethics community at the institutional, national, and international levels; seizing opportunities to engage in research ethics or to promote research ethics; and conducting empirical research on research ethics are all inputs integrated into the crucial posttraining process of being a proactive participant in the research ethics community. Trainees can become proactive participants by attending or conducting training sessions, participating in committees, and promoting research ethics principles through lectures and publications. On the other hand, program faculty recommend that trainees attend international meetings and sit in on data safety boards or other similar committee meetings. Another posttraining input is the JHF program biannual reunion meeting, which is the congregation of all JHF alumni and faculty for the purposes of networking, idea exchange, and general capacity development. As a result, some success comes from trainees' own hard work to get research ethics work established in their countries; other success comes because, through the high visibility of the program, faculty are asked to recommend trainees for speaking, training, and consulting assignments.
The outputs of the posttraining stage are specific to each trainee and are, in part, a reflection of his or her own initiative seen through research accomplished, workshops conducted, and active networking. The outcomes are also a function of the posttraining professional context, individual to each trainee; for example, some trainees return to institutions that have an IRB, whereas others may be creating the first IRB in their area. In one example of a posttraining outcome, a trainee from Zimbabwe has been invited to more than 30 workshops. In another, a trainee from the Democratic Republic of Congo has helped establish two IRBs there. All the JHF trainees have been actively involved in teaching, training, and nationally significant research ethics work since returning to their home countries. For example, a trainee from Nigeria has initiated a training program in bioethics in his institution. Invitations to participate in conferences and workshops, consultation for specific research studies, and participation in regional networks are common for trainees in the posttraining stage. The initial set of publications by the trainees is also appearing in the literature slowly.16 One of the trainees has gone on to enroll in a masters program in bioethics in North America, and another is enrolled in a PhD program in research ethics in Africa.
The enduring outcomes of this work will be institutional changes made in policies, guidelines, and awareness to support research ethics in Africa. Ultimately, the culmination of all of the phases of our program, together with other similar efforts, will result in a contribution to increase research ethics capacity in Africa.
Eight years ago, international research ethics made front-page news. The debate that prompted the publicity—how perinatal HIV-transmission trials should be designed in developing countries—received enormous attention, with articles in the lay press, editorials in scientific journals, and e-mail discussion groups arguing about placebos, standards of care, and international justice. Such debates must be considered useful for the field, because the international research community was forced to focus increased attention on ethics. However, the most troubling aspect of the debate was that, with a few exceptions, it occurred between and among developed-world researchers and commentators about the lives, needs, and cultural expectations of those from developing countries.
The NIH FIC's initiative to train scholars from the developing world in bioethics is one response to this problem, developing capacity in research ethics and enabling developing countries to have a voice in these matters that directly affect them. The JHF program is among the longest-standing of these FIC awards. African professionals trained in research ethics can act as informants and advocates for their own countries in these debates, asserting a point of view that was conspicuously absent before. Although the formal training of the JHF program is provided in the United States, there is a required practicum project to be carried out in the trainees' home countries. Our program design allows these trained professionals to be a resource and a powerful local voice in their own countries' research ethics debates.13 It has been a challenge to create, develop, and implement an effective training program for professionals, and the program has required adjustments throughout its duration to improve and strengthen its impact on the trainees.
Successes of JHF
In this article, we have attempted to present an initial assessment of the JHF program. The JHF training program has been successful in providing research ethics education to trainees and motivating them to contribute to the field of research ethics in Africa. We accomplished this by carefully selecting trainees from the beginning—a critical step in the success of any training program. The trainees' countries of origin are shown in Table 4, demonstrating that our trainees come from low-income countries in Africa with huge health problems. The curriculum with its required courses has been essential in providing the trainees a foundational knowledge of bioethics. The weekly seminars provide an essential forum for the trainees to discuss not only ethical issues but also general logistics of the program and ideas for their practicum. The faculty and affiliated professionals of diverse disciplines are great resources for the trainees during their time at JHU and through their continued pursuits in research ethics.
An obviously important aspect of the training program is the learning environment. Simply being in the United States during part of the training offers an abundance of different opportunities to the trainees, and being situated at JHU enhances the amount and quality of these opportunities by providing the trainees access to the most up-to-date resources for their training. This new setting offers trainees dependable Internet access, extensive libraries, and innumerable international research projects in Africa. The teaching style used in the JHF training program is not only content rich; it is rich in process as well. The program begins with a focus on classroom learning, which offers a considerable amount of guidance. Then, research methodology is introduced, which opens the door to research opportunities and investigations. Research under mentorship is a form of learning that maintains moderate amounts of guidance but encourages independent thought and work. Learning how to conduct high-quality research is an essential step towards the practicum. Finally, the practicum is a learning experience that focuses on the opportunity for the trainee to demonstrate what he or she has learned. The gradual progression of guided work to independent work integrated into the curriculum is key to our goal of developing research ethics capacity in the trainees.
During the past four years, adjustments have been made to incorporate new elements into the program. These adjustments were a result of lessons learned in the program and recommendations from trainees. During four years of the training program, the number of applicants increased dramatically. To respond to this increase, we added a new prescreening step to the application-review process, in which we checked that the application was complete, that the applicant spoke English, and that the applicant had obtained at least a bachelor degree. We have also adjusted the application due date to fall two months earlier, to allow adequate time for visa processing; this was necessary because of recent political turmoil in sub-Saharan Africa and security concerns. Not only has the program been flexible to the needs of the trainees and faculty, but it has also changed to account for changes in the world. In addition, the content of methodology seminars was also adjusted each year according to the trainees' experience and learning objectives.
Lessons learned by JHF
We have learned some important lessons during the four years of the program, especially concerning posttraining activities. We learned the importance of maintaining regular contact with trainees on their return to their home countries. The program lost contact with only one trainee during the posttraining period, and that was determined by the trainee's decision to pursue work independently of the program. To ensure a timely, comprehensive completion of the practicum, we realized that financing of the practicum should occur in stages and must be linked to interim objectives of the practicum. We also learned that we needed a standard way to observe trainees' progress as they completed program requirements, so we created a monthly progress report form to actively monitor trainees' work and maintain contact with them in the posttraining stage. These elements are crucial to the successful completion of the practicum and the continued involvement of trainees in research ethics in Africa. This communication is equally important to the global problem of bridging the information gap between developed and developing countries.11 These changes have increased the efficiency of the program and the proficiency of trainees.
Areas for continuing improvement
There are still some issues in the JHF that are unresolved. For example, in the selection process, it is uncertain whether it is more beneficial to the development of research ethics capacity in Africa to choose trainees who are senior and established in a career, or more junior candidates whose career paths are more flexible. Senior applicants are more experienced and established in positions where they can make an impact immediately, whereas junior trainees still have the opportunity to devote their careers to research ethics entirely, rather than adding it to their numerous preexisting commitments. At this point, it is unclear who will have more of an impact on research ethics in Africa. Another unanswered question about training is the effectiveness of didactic versus practical training and whether one should be emphasized more than the other. One factor we may explore in regard to this debate is the social impact made through each method of teaching.17
We could also improve the JHF program by increasing faculty members' awareness of the social and political issues of the trainees' countries as well, because without a certain awareness of these problems, it is difficult to develop a strong research ethics framework to be applied there.18 We have also reviewed the role of external faculty, because it has not always fulfilled the desired goal of close mentoring, often because of logistical or communication problems. Another area of uncertainty for the program is whether to offer a degree. With the cost of tuition, this would reduce the number of trainees to about 25% of the program's current capacity, to be financially viable. However, offering a degree would allow some trainees to enter another degree program if they were competent and promising.
Current published literature does not offer a comprehensive framework for evaluating efforts toward developing research capacity or ethics capacity.19 The lack of agreed indicators and processes of evaluation means that the JHF program has had to develop its own method, as described above. This has the advantage of being innovative and specific to the program, yet it does not allow cross-program comparability. We hope to stimulate such a dialogue across training programs in research ethics with this article.
At this time, and on the basis of a single program, it is not appropriate to evaluate the impact on research ethics capacity development in Africa. However, this case study makes it possible to assess the meaning and use of research ethics training programs. Our outputs give a general indication of the pending success of research ethics capacity development progress for Africa. The JHF program has helped provide one type of foundation for a research ethics network in Africa. The immediate outputs—the trainees—have raised awareness of research ethics and initiated some action in Africa. The anticipated outcome is that the awareness will spread and that research ethics initiatives will become more prevalent, reducing exploitation of research participants. A successfully developed awareness of research ethics issues will produce researchers with moral commitments supported by a strong policy environment.5 Eventually, the contribution of all research ethics training programs to capacity development in Africa will result in the establishment of a self-contained and strong research ethics system in Africa.
1 Lurie P, Wolfe SM. Unethical trials of interventions to reduce perinatal transmission of the human immunodeficiency virus in developing countries. N Engl J Med. 1997;337:853–856.
2 Varmus H, Satcher D. Ethical complexities of conducting research in developing countries. N Engl J Med. 1997;337:1003–1005.
3 Sharpiro HT, Meslin EM. Ethical issues in the design and conduct of clinical trials in developing countries. N Engl J Med. 2001;345:139–142.
4 Kass N, Hyder AA. Attitudes and experiences of U.S. and developing country investigators regarding U.S. human subjects regulations. Washington, DC: National Bioethics Advisory Commission; 2001. Commissioned paper B-3.
5 Lavrey J. A culture of ethical conduct in research: the proper goal of capacity building in international research ethics. Geneva, Switzerland: Commission on Macroeconomics and Health; 2003. CMH working paper WG2:5, 3-4.
6 Nuffield Council on Bioethics. The ethics of clinical research in developing countries. London, UK: Nuffield Council on Bioethics; 1999.
7 National Bioethics Advisory Commission (NBAC). Draft report on ethical issues in international research. Washington, DC: National Bioethics Advisory Commission; 2000.
8 Nuffield Council on Bioethics. The ethics of research related to healthcare in developing countries. London, UK: Nuffield Council on Bioethics; 2002.
9 Ayle HD. Curbing the global AIDS epidemic. N Engl J Med. 2003;348:1802–1805.
10 De Cock KM, Mbori-Ngacha D, Marum E. Shadow on the continent: public health and HIV/AIDS in Africa in the 21st century. Lancet. 2002;360:67–72.
11 Horton R. North and South: bridging the information gap. Lancet. 2000;355:2231–2236.
12 Equinet Steering Committee. Equity in health in Southern Africa: turning values into practice. In: R. Loewenson, ed. Regional Conference “Building Alliances for Equity in Health.” Midrand, South Africa: Regional Network for Equity in Health in Southern Africa; 2000;1–29.
13 Pang T. Brain drain and health professionals. BMJ. 2002;324:499–500.
16 Tangwa GB. Between universalism and relativism: a conceptual exploration of problems in formulating and applying international biomedical ethical guidelines. J Med Ethics. 2004;30:63–67.
17 Smith R. Measuring the social impact of research. BMJ. 2001;323:528.
18 Danish Agency for International Development. DANIDA's bilateral programme for enhancement of research capacity in developing countries (ENRECA). Denmark: DANIDA, 2000. [No. 200/5].
19 Hyder AA, Akhter T, Qayyum A. Capacity development for health research in Pakistan: the effects of doctoral training. Health Policy Plan. 2003;18:338–343.