Kalichman, Michael W. PhD; Plemmons, Dena K. PhD
Responsible conduct of research (RCR) education is required for some recipients of federal research funding, but this is not in itself an argument that those RCR programs must be effective. The roots of a rationale for effective RCR programs can be found in the scientific community, which has frequently made it clear that integrity of science depends on the integrity of scientists.1–4 Because scientific integrity consists of much more than not engaging in falsification, fabrication, or plagiarism, it is reasonable to assume that standards of responsible conduct must somehow be taught and learned in the practice of research. However, the evidence is that attention to these standards is rarely part of the informal, mentoring relationships between established researchers and trainees.5,6 One alternative to such informal socialization is formal training in RCR, but little is known about the effectiveness of strategies to promote RCR,4,7 perhaps in part because the goals of RCR education are neither clearly articulated nor widely accepted.
A prerequisite for any assessment of the efficacy of training in research ethics is a clear definition of goals and objectives. However, this information is not generally available for individual courses, and it has certainly not been summarized for the wide range of instructional activities provided within the field. When Mastroianni and Kahn8 reviewed National Institutes of Health (NIH) training grants at 45 institutions in 1996, they found that few of the training programs identified RCR education goals and objectives in their curricular material. As they concluded, “this would seem to imply that programs do not approach training in the same manner they would typically approach other educational efforts.” The purpose of our study was to assess the existing landscape of goals for RCR education by interviewing instructors of RCR courses.
For this study, reviewed and approved by the University of California–San Diego institutional review board (project #040541SX), we identified recipients of new NIH training grants awarded in 2000 using the NIH Computerized Retrieval of Information on Scientific Projects (CRISP) database.9 In 2002 and 2003, we contacted all 116 training grant recipients by e-mail to request contact information for the instructors of their NIH-required courses in RCR. The RCR instructors were solicited by e-mail to participate in interviews, which were subsequently scheduled, usually by e-mail. In 2003 and 2004, interviews of the instructors were conducted by telephone for all but one, who was interviewed in person. The questions were sent in advance by e-mail and repeated verbally for the interview. All but four of the interviews were successfully recorded on tape and transcribed. The duration of the 46 recorded interviews ranged from 7.3 to 34.8 minutes, and was an average of 17.9 minutes.
We developed and refined the questionnaire (Appendices 1 and 2) through informal communication with a convenience sample of subject matter experts, all of whom were identified because of their national recognition as experts in RCR instruction, and focus-group discussions with individuals identified because of their experience or interest in RCR teaching. For the initial informal survey, 22 RCR instructors were asked to “… briefly summarize your goals, objectives, or intended outcomes for teaching biomedical research ethics.” The resulting responses from 20 instructors were tabulated and coded separately by three individuals. The scorers reviewed their respective categorizations of the possible responses and reached consensus on a range of possible themes corresponding to goals for teaching RCR. These possibilities were then consolidated into a series of questions to guide focus-group discussions with 17 individuals. On the basis of the focus-group discussions, a final survey instrument was completed and used for the subsequent interviews of RCR instructors.
The final survey consisted of four forced-choice demographic questions about the respondent’s institution, the respondent’s teaching experience in research ethics, and characteristics of the audiences for RCR courses. The remainder of the survey included forced-choice and open-ended questions about instructional goals across five domains: knowledge, skills, attitudes, behaviors, and community.
After data collection, we conducted a primary analysis of the numerical responses for demographic questions and the items ranked for the categories of knowledge and skills. Data were entered into a database (Microsoft Excel), and numerical analyses consisted of counts and percentages. Other than the demographic data, these quantitative data and other more detailed results for each of the categories of knowledge, skills, attitudes, behaviors, and community will be reported separately in other manuscripts. The focus of this article is on the demographic questions and an overview of the responses to open-ended questions about goals for teaching RCR.
E-mail messages were sent to the principal investigators (PIs) of all 116 training grants to request contact information for the person(s) responsible for providing the required RCR training. From a total of 116 possible PIs, 22 (19%) did not respond at all, and messages for two of the PIs were returned as undeliverable. Although six of those contacted indicated that they were no longer the PIs of the identified training grants, four provided the names of the new PIs and one provided contact information for the RCR instructor. No RCR instructors were identified in seven cases: two PIs indicated that the required RCR training was provided on Web sites with no instructor, one defined the RCR training format as a proseminar that involved several faculty, and four indicated that the corresponding grant did not include an RCR training component because the program was too short, not appropriate for RCR instruction, or not a training grant. Thus, out of 92 individuals we successfully contacted by e-mail, 84 (91%) identified RCR instructors for their grants.
The initial e-mail survey yielded 115 instructors for the required research ethics instruction component of 84 training grants (several PIs identified more than one instructor at their institution). We sent e-mail requests to take part in an interview to all 115 identified RCR instructors: three e-mails were returned as undeliverable; 23 instructors did not respond despite up to three e-mail requests; 21 responded that they were not correctly identified as RCR instructors; one responded that he/she was not currently an RCR instructor; eight initially responded but did not respond to follow-ups to schedule the interview; and nine responded but declined to be interviewed because of time constraints, sabbatical schedules, or, in one case, questions about the content of the survey. The remainder (n = 50) were successfully interviewed. For those verified as RCR instructors (n = 67), this corresponds to a 75% response rate.
The 50 instructors interviewed represented 37 different institutions. Although no questions were designed to address the format of the courses taught, it was clear from the survey that at least some respondents were course directors, but not necessarily instructors and, at least in one case, the course was not a classroom course, but a series of Web-based modules. Most instructors (n = 32; 64%) were from large, public, not-for-profit research institutions. When asked to characterize their institutions as research, teaching, and/or clinical, 49 (98%) selected research, 44 (88%) teaching, and 36 (72%) clinical. Forty-eight (96%) characterized their institution as large and only two (4%) as small. When asked whether their institutions were public or private, 31 (62%) said public, 17 (34%) private, and two (4%) a mixture of public and private. Only one respondent identified his or her institution as for profit, whereas 47 (94%) selected not for profit, and two did not answer. Independently using the definitions of the 2000 Carnegie Foundation classifications,10 the institutions of the interviewees were predominantly doctoral/research universities-extensive*(n = 41; 82%); most of the remaining institutions were medical schools and medical centers* (n = 8; 16%); and one was a nonprofit research institution (n = 1; 2%) not included in the Carnegie 2000 classifications.
Teaching experience varied widely among the respondents (Figure 1), but the median number of RCR courses taught was nine. Only one individual had not yet taught an RCR course, and another reported having taught approximately 400 research ethics courses, workshops, or seminars.
The extent to which instruction was required varied considerably. Only a few instructors (n = 6; 12%) reported that RCR instruction was required for all trainees in their institutions, irrespective of their source of funding, as recommended by current NIH policy.11 Only one of these six institutions reported that instruction was required, not just for trainees (graduate students and postdoctoral fellows), but also for faculty. Eleven additional instructors explicitly noted some form of requirement for all graduate students, but not other trainees, to receive training in their RCR courses. In addition, 20 (40%) of the instructors noted that their courses were required both for NIH trainees and for some other group or groups at their institutions. Only two respondents (4%) identified their courses as being required only for NIH trainees. Conversely, despite having been identified as the instructor for an RCR course to meet the NIH training grant requirement, eight instructors (16%) reported that attendance in their courses was entirely voluntary and that the courses were not intended to meet NIH training grant requirements; three other instructors (6%) indicated that their courses were required for particular trainees, but not those on NIH training grants.
Instructors were asked to assess the importance of a wide range of possible goals for RCR instruction and to propose other goals that should be considered important. The details of these responses are too extensive for a single report and are being prepared for separate publications (Plemmons and Kalichman, manuscripts in preparation). However, three overall findings were particularly noteworthy.
First, the instructors’ ratings of the importance of possible goals for RCR instruction varied markedly. This variation was reflected in the range of open-ended responses to questions about the purpose of their own courses. Some instructors were quite ambitious in their goals to
decrease the amount of unethical behavior going on in the world.
[teach students to] demonstrate commitment, responsibility, and respect for all aspects of research, including subjects of research, whether they’re animal models or human models, as well as those that would be affected by the results of the research.
Others were more modest in their objectives, aiming to
[teach students] to recognize what research misconduct is and to avoid it.
keep them out of trouble.
[teach] better research record keeping.
Some instructors were focused on transmitting knowledge and information to
make them aware of policies and regulations.
know where to go for information.
and others were more interested in teaching trainees the skills to
(1) recognize when they have an ethical problem, (2) know how to analyze the problem, (3) come up with strategies for resolving it.
identify the stakeholders in any ethical problem, [and] what those people have to gain or lose based on different decisions that are made.
Other instructors sought to change trainees’ attitudes to:
encourage a sense that part of doing good science is through fulfilling ethical principles, that those two things go hand in hand, as opposed to being in opposition.
adopt the attitude that issues related to RCR are not remote or occult, or not an integral part of their lives as scientists. You can’t avoid it. Duty is thrust upon you simply by virtue of your position. That attitude ought to confer responsibility.
or shape trainees’ behavior to
act in a way that embodies the insights they’ve gotten in the course in terms of integrity, including having the wherewithal to resist social pressures, institutional pressures, financial pressures, to cut corners.
adhere to the standards and principles. They should not have just an awareness, but behavior that fosters integrity and professionalism.
see that individuals have the will and the courage to follow through on what it takes to act ethically.
Examples of goals explicitly noted for knowledge, skills, attitudes, behaviors, and community are summarized in List 1.
Second, it was clear that even when explicitly asked about their goals for teaching RCR courses, some respondents did not readily distinguish between goals and methods. For example, some of the instructors described their “goals” only in terms of the methods that might be used to achieve them:
Anonymous case study examples of the difference between misconduct and differences in style or personality challenges.
Open roundtable discussions versus formal lectures.
Third, some but not all of the instructors clearly took the responsibility of RCR instruction very seriously. Their commitment was reflected in the depth of their answers to questions about their motivations for teaching RCR courses. For example:
[I] want students to act ethically, to have a deeper understanding of ethics, rather than simply following narrowly prescribed steps.
A broadening of their attitudes, understanding, and sensitivity for what kinds of things should be considered when making these kinds of decisions.
The primary finding of this study was the wide diversity of instructors’ goals and understanding of goals for the teaching of RCR. Although such a finding is not necessarily surprising, it is significant because of what it tells us about the state of RCR education. Presumably, RCR education is required by some federal agencies,11,12 institutions (e.g., University of Minnesota and Duke University), and individual graduate programs because of the perception that such education will benefit individual trainees as well as the scientific enterprise broadly. However, if the goals of RCR instructors are so highly variable, then it is unlikely that their diverse efforts at RCR education will result in a common set of outcomes. Although there may be value in diversity of goals for any complex educational activity, it is important to clarify both what are and what should or can be our common goals.
A few of the general findings of this study raise questions about who is teaching RCR. Several of the respondents had real difficulty answering basic questions about the importance of particular goals for teaching RCR as well as articulating their own goals. Assuming that effective educational experiences depend first on being clear about educational objectives, it is worrisome if RCR instructors are unclear about their aims. Some participants failed to recognize the basic distinction between the goals for teaching RCR and the methods used to achieve those goals. Others seemed to struggle with the distinction between the goals of conveying new knowledge and the goals of teaching new skills. Finally, many instructors viewed their goals with a level of certainty that would have been hard to maintain if they were aware of other equally confident instructors reporting their own, very different views.
Although we did not address the specific question of who teaches RCR, these findings suggest that future assessments of RCR education need to better characterize the instructors providing RCR education in training grants. Key questions include: To what extent have RCR instructors been taught about teaching, and in particular adult education? What are the disciplinary backgrounds of typical RCR instructors? Do they have formal education in research ethics? Do they have experience in experimental research? Are they themselves active researchers, ethicists, philosophers, social scientists, and/or administrators? To what extent have they interacted with other RCR instructors? To what extent are they aware of and do they make use of available resources for RCR instruction? And how do they select their topics, materials, and instructional formats?
Although this study was not designed to assess compliance with the NIH training grant requirement for RCR instruction, the results suggest that a high percentage of training grant directors do not actually know who is providing RCR instruction to their trainees. Of the 89 individuals identified by the training grant directors as RCR instructors whom we successfully contacted, 21 reported that they were not RCR instructors, and one was no longer an RCR instructor. Thus, 25% of the individuals identified by the training grant directors as the person responsible for providing RCR instruction were not, in their views, doing so. Even more surprising, 11 of the 50 instructors interviewed explicitly reported that the participants in their courses were not NIH trainees. Thus, 22% of the instructors identified as being responsible for meeting the training grant requirement for RCR instruction apparently did not see—or at least did not know that they were seeing—the relevant NIH trainees in their courses.
This study was also not designed to assess how frequently RCR instruction is required for anyone other than trainees supported by NIH training grants. However, the findings suggest that where RCR instruction exists, it is not widely required. RCR education is only seldom required of everyone in a department or institution (i.e., as reported by only 1 of 50 instructors), and only six (12%) of the 50 instructors reported that RCR instruction was required for all trainees, as was recommended by the NIH training grant requirement11: “[p]lans that incorporate instruction in the responsible conduct of research for all graduate students and postdoctorates in a training program or department, regardless of the source of support, are particularly encouraged.” It is heartening that 40 (80%) of those interviewed reported that training was required for at least some group other than NIH trainees, but it is worth asking whether this apparent good news is good enough. Even setting aside the desirability of providing RCR instruction to research faculty and staff, 88% of the instructors interviewed reported that RCR instruction was required for some, but not all, research trainees in their institutions. If these reports are true, then the clear underlying message is that despite the NIH mandate, RCR education is neither included in nor viewed as part of basic research training. The implicit question is, therefore, if responsible conduct is essential to being a good scientist, then how are the relevant knowledge, skills, attitudes, and behaviors to be taught, and by whom?
Our study design does have limitations. First, the purpose of this initial study was to find RCR instructors at many different institutions to assess instructional goals independent of any particular course format—the sampling method was not designed to find similar types of courses. Second, identifying instructors’ self-reported goals is not the same as having measured goals or assessed outcomes. We did not ask about the method(s) of assessment that instructors used to evaluate their students’ performance, or whether the goals that they articulated were or could be met. A different type of study is needed to know the extent to which the goals noted in the domains of knowledge, skills, attitudes, or behaviors are actually met (or even feasible). Finally, the purpose of this study was to assess existing goals for teaching RCR, not ideal goals. Therefore, the resulting data do not in any sense define what should be done in RCR education, but only what current instructors report to be their own goals or goals that they deem important. Although these data provide a necessary foundation, a different approach is needed to develop consensus among knowledgeable teachers of RCR.
Required, formal instruction in research ethics is relatively new to most training programs, and this requirement is met in many different ways. Clearly, supporters of RCR education intend that research ethics instruction should be effective. It is essential, then, that we have a clear sense of what the goals of such instruction should be, and that we identify measurable instructional outcomes. Researchers studying the effectiveness of RCR education should also remain cognizant that different approaches to teaching may lead to different outcomes, whatever the goals. A next step in such research is to measure the success of different approaches in existing courses by measuring their outcomes.
Drs. Carole Roth and Liberty Smith are thanked for their invaluable contributions to the design and support during the earliest stages of this project, including tabulation and coding of responses from the convenience sample of RCR instructors, preparation for and conducting the focus-group discussions to finalize the survey instrument, and conducting the two initial interviews. Andrea Paik is thanked for conducting the remaining 48 interviews and for thoughtful editing of the final manuscript.
The research described in this manuscript was supported by NIH grant AI001591.
The views expressed are those of the authors and are not necessarily shared by the acknowledged individuals.
1 Institute of Medicine. The Responsible Conduct of Research in the Health Sciences. Committee on the Responsible Conduct of Research, National Research Council. Available at: (http://www.nap.edu/books/0309062373/html
). Accessed May 30, 2007.
2 Commission on Research Integrity. Integrity and Misconduct in Research. Washington, DC: Department of Health and Human Services; 1995.
3 National Academy of Sciences, Panel on Scientific Responsibility and the Conduct of Research, National Academy of Sciences, National Academy of Engineering, Institute of Medicine. Responsible Science, Volume I: Ensuring the Integrity of the Research Process. Available at: (http://www.nap.edu/books/0309047315/html
). Accessed May 30, 2007.
4 Institute of Medicine, Board on Health Sciences Policy and Division of Earth and Life Studies, Institute of Medicine and National Research Council of the National Academies. Integrity in Scientific Research: Creating an Environment that Promotes Responsible Conduct. Available at: (http://www.nap.edu/books/0309084792/html
). Accessed May 30, 2007.
5 Brown S, Kalichman MW. Effects of training in the responsible conduct of research: a survey of graduate students in experimental sciences. Sci Eng Ethics. 1998;4:487–498.
6 Swazey JP, Anderson MS. Mentors, advisors and role models in graduate and professional education. In: Rubin ER, ed. Mission Management. Vol. 2. Washington, DC: Association of Academic Health Centers; 1998:165–185.
7 Powell S, Allison MA, Kalichman MW. Effectiveness of a short-term course in the responsible conduct of research for medical students. Sci Eng Ethics. (in press).
8 Mastroianni A, Kahn J. Encouraging accountability in research: a pilot assessment of training efforts. Account Res. 1999;7:85–100.
9 CRISP (Computer Retrieval of Information on Scientific Projects). Available at: (http://crisp.cit.nih.gov
). Accessed May 14, 2007.
10 Carnegie Foundation. Carnegie Foundation’s Classification of 3,941 Institutions of Higher Education. Facts and Figures. Available at: (http://chronicle.com/stats/carnegie
). Accessed May 14, 2007.
11 National Institutes of Health. Reminder and update: requirement for instruction in the responsible conduct of research in national research service award institutional training grants. NIH Guide Grants Contracts. 1992; 21:43.
12 National Science Foundation. Question 8.a: What is expected for the training in the responsible conduct of research. In: Frequently Asked Questions (FAQ) about the Integrative Graduate Education and Research Training (IGERT) Program. Available at: (http://www.nsf.gov/pubs/1998/nsf98116/nsf98116.htm
). Accessed May 14, 2007.
*Doctoral/research universities-extensive: “These institutions offer a wide range of baccalaureate programs and are committed to graduate education through the doctorate. They award 50 or more doctoral degrees per year across at least 15 disciplines.” Medical schools and medical centers: “These institutions award most of their professional degrees in medicine. In some instances, they include other health-professions programs, such as dentistry, pharmacy, or nursing.”10 Cited Here...